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Preface 


This  casebook  provides  concrete  (fictional)  examples  that  demonstrate  many  aspects  of 
how  addictions  programs  can  use  evaluation  techniques  (ranging  from  very  simple 
questions  to  fairly  large  scale  evaluations)  to  address  quality  enhancement  questions. 
The  assumption  is  that  people  can  learn  about  evaluation  by  reading  concrete  examples 
of  evaluations  that  have  been  used  in  programs  similar  to  the  particular  reader ^s 
program  of  concern.  For  example,  a program  manager  of  an  addictions  treatment 
program  might  first  examine  some  of  the  treatment  examples.  The  first  concern  in 
designing  the  casebook  was  that  there  he  a range  of  programs  described  so  that  each 
reader  can  find  an  example  using  a program  similar  to  the  program  with  which  he  or 
she  is  involved. 

Another  concern  was  that  each  level  of  evaluation  be  illustrated  by  one  or  more 
examples.  In  addition,  it  was  intended  that  the  examples  include  a range  of  levels  of 
difficulty  and  resource  needs.  These  are  indicated  on  the  first  page  of  each  section  of 
the  example.  Finally,  the  examples  are  organized  so  that  the  sections  within  them 
usually  proceed  from  the  less  complex  to  the  more  complex. 

Many  methodological,  measurement  and  ethical  issues  are  covered  throughout  the 
casebook.  Specific  issues  can  be  located  by  using  the  indices  in  Part  III.  Major 
methodological  or  other  issues  that  arise  in  a particular  example  are  described  more 
fully  in  a box  following  the  example.  The  casebook  can  be  used  both  by  browsing 
through  examples  of  interest  and  by  using  the  indices  to  find  examples  of  particular 
kinds  of  evaluation. 

It  is  hoped  that  the  range  of  case  examples  will  illustrate  both  the  opportunities  and 
difficulties  of  program  evaluation.  Examples  are  provided  to  show  that  some  apparently 
simple  quality  enhancement  questions  may  require  expensive  and  sophisticated 
evaluation  methods.  Alternate  examples  illustrate  that  it  is  possible  to  address  other 
useful  quality  enhancement  questions  using  simple  and  inexpensive  procedures.  Thus, 
the  collection  of  examples  should  convey  both  the  humbling  and  the  exciting  aspects  of 
program  evaluation. 
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CAUTION:  ALL  CASE  EXAMPLES  ARE  FICTIONAL  ! 


All  case  examples  provided  in  this  casebook  are  fictional  and  do  not  reflect 
real  findings  from  evaluations  of  addictions  programs.  In  some  cases,  the  idea 
for  the  example  came  from  a real  evaluation.  However,  in  all  cases,  methods 
and  results  were  altered  in  order  to  present  evaluation  techniques  as  clearly  as 
possible. 

We  would  like  to  acknowledge  the  following  reports  that  were  used  as  sources  of  ideas  for  some 
examples,  but  we  must  emphasize  that  the  related  examples  do  not  necessarily  reflect  either  the 
methods  actually  used  in  these  studies  or  the  results  found  in  them. 
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IX 


I.  EVTRODUCTION 


How  are  program  evaluation  and  quality  enhancement  related? 

Program  evaluation  is  the  term  used  to  describe  methods,  designs  and  theories  that 
are  used  to  evaluate  interventions  (i.e.,  programs)  in  real-life  contexts.  Classical 
program  evaluation  emerged  primarily  out  of  education  and  psychology  and  is 
rooted  in  quantitative  social  science  methods.  Quality  theory  and  practices  (i.e., 
quality  assurance,  quality  improvement,  quality  enhancement)  are  related  to 
program  evaluation,  but  emerged  mostly  from  business  and  management.  Recent 
thinking  in  quality  enhancement,  however,  indicates  a merging  of  program 
evaluation  techniques  within  quality  enhancement  structure  and  questions. 

Quality  enhancement  focuses  on  identifying  specific  system  problems,  collecting 
relevant  data,  and  using  data  to  identify  solutions  for  the  problems.  As  will  be 
explained  in  the  following  sections,  program  evaluation  techniques  can  be  useful 
at  each  stage  of  the  quality  enhancement  process. 


Program  evaluation  includes  more  than  evaluating  program 
effectiveness. 

Many  people,  when  they  think  "program  evaluation",  think  of  one  question  "Is  this 
program  effective?"  In  the  additions  field,  "effectiveness"  is  usually  taken  to  mean 
that  a prevention  program  can  actually  be  shown  to  prevent  alcohol  and  drug 
problems  and  that  people  attending  a treatment  program  actually  stop  using  alcohol 
and  other  drugs.  However,  evaluating  the  ultimate  effectiveness  of  a program  is 
rarely  an  easy  process.  For  programs  delivering  human  services,  the  following 
questions  need  to  be  addressed  in  order  to  demonstrate  that  the  program  is 
effective: 

• Do  those  who  receive  the  program  achieve  better  outcomes  than 
those  who  do  not  receive  the  program?  (For  example,  is  there  a 
higher  rate  of  reduced  drinking  among  clients  who  attend  an 
addictions  treatment  program  than  among  a comparable  group  of 
people  who  have  drinking  problems  but  who  do  not  attend  the 
program?  Do  school  children  who  receive  alcohol  and  drug 
education  develop  fewer  alcohol  and  drug  problems  in  later  life  than 
a comparable  group  of  children  who  do  not  receive  alcohol  and  drug 
education?) 


• Why  do  people  who  attend  the  program  achieve  better  outcomes  than 
those  who  do  not?  (i.e.,  If  the  program  consists  of  many  services, 
which  services  are  effective?  If  people  provide  the  service,  to  what 
extent  do  outcomes  depend  on  the  skills  or  personality  of  the  service 
providers?) 

• Is  the  program  100%  effective  for  all  who  receive  it?  If  not,  do 
some  kinds  of  people  benefit  more  than  others? 

In  sum,  the  question  "Is  this  program  effective?"  actually  includes  at  least  the 
following  questions: 

• Is  this  program  more  effective  than  no  program?  A different 
program? 

• Which  aspects  of  the  program  are  effective? 

• For  whom  is  the  program  effective? 

Clearly,  except  for  very  simple  programs,  addressing  the  question  "Is  this  program 
effective?"  can  be  a costly  and  difficult  (sometimes  impossible)  task.  When 
evaluation  is  defined  so  narrowly,  it  is  unlikely  to  be  used  by  programs  except  in 
rare  circumstances  where  research  and  other  resources  are  available.  However, 
as  already  stated,  program  evaluation  includes  methods  and  designs  that  can  be 
used  to  address  a variety  of  evaluation  questions,  not  just  overall  effectiveness. 

For  example,  evaluation  techniques  provide  methods  for  framing  quality 
enhancement  questions,  for  collecting  data  to  address  quality  enhancement 
questions,  and  for  testing  possible  solutions  to  quality  enhancement  questions.  For 
example,  one  technique  of  program  evaluation  is  "rationale  assessment".  This 
procedure  examines  the  assumptions  of  a program  or  system  - why  certain  things 
are  done  and  how  these  relate  to  the  program’s  objectives.  The  process  of 
systematically  examining  the  rationale  of  a program  can  identify  potential  problems 
that  form  the  basis  of  quality  enhancement  questions.  Program  evaluation  also 
includes  monitoring  implementation  of  programs.  This  approach  can  be  used  to 
collect  data  to  address  specific  quality  enhancement  questions  as  well  as  to  identify 
new  questions.  Finally,  evaluation  also  includes  approaches  to  measuring  costs 
and  outcomes,  both  of  which  are  relevant  to  many  quality  enhancement  questions. 


Types  of  questions  leading  to  program  enhancement 

Human  services  programs  (including  addictions  programs)  tend  to  change  or 
evolve  over  time.  Consequently,  program  evaluation  should  be  part  of  the  ongoing 
process  of  quality  enhancement,  where  information  is  collected  and  used  to 
advance  the  program  towards  its  goals.  Examples  of  program  evaluation  questions 
posed  as  part  of  quality  enhancement  include: 

• Does  our  program  contain  all  the  parts  necessary  to  achieve  our 
program  goals? 

• Does  the  program  address  the  major  needs  of  its  clients/consumers? 

• Is  the  program  serving  the  clients/consumers  it  was  intended  to 
serve? 

• What  proportion  of  clients/consumers  who  participate  in  the  program 
achieve  positive  outcomes? 

• In  what  areas  do  the  most  clients/consumers  change? 

• What  services  are  provided  to  each  client/consumer? 

• Does  the  program  make  use  of  other  community  resources?  Do  other 
community  resources  make  appropriate  use  of  the  program? 

• Which  of  several  acceptable  ways  of  delivering  service  costs  the 
least? 


Who  asks  quality  enhancement  questions? 

Anyone  involved  with  a program  may  think  up  a question  that  could  lead  to 
collecting  evaluation  information  useful  for  enhancing  the  program.  The  program 
manager  may  be  the  most  likely  person  to  raise  such  questions,  because  he  or  she 
is  responsible  for  optimizing  the  operation  of  the  program.  However,  questions 
pertaining  to  program  quality  can  come  from  a variety  of  sources  including: 

• program  manager 

• program  staff 

• program  clients/consumers 

• program  board 

• program  funder 

• other  programs 

• community  groups 

• researchers. 
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II.  EVALUATION  FRAMEWORK  USED  IN  DEFINING 
QUALITY  ENHANCEMENT  QUESTIONS 


To  provide  a simple  organization  for  asking  quality  enhancement  questions,  the 
following  framework  of  evaluation  levels  was  adopted: 

• Needs  Assessment 

• Logic  Model 

• Client/Consumer  Characteristics 

• Program  Delivery 

• Outcome  Evaluation 

• Client/Consumer  Satisfaction 

• Economic  Evaluation. 


These  levels  have  been  adopted  as  those  most  likely  to  be  useful  for  program  staff 
and  others  who  are  not  experienced  evaluators  who  want  to  begin  to  use 
quantitative  evaluation  techniques  to  address  quality  enhancement  questions.  The 
following  describes  the  kinds  of  questions  that  might  be  addressed  at  each  level. 

• Needs  Assessment 

- Is  there  a need  for  this  kind  of  service  in  this  community? 

- What  are  the  characteristics  of  people  who  might  use  the 
service? 

- How  many  beds/staff/other  facilities  are  likely  to  be  needed? 

• Logic  Model 

- What  are  the  components  of  the  program? 

- What  are  the  objectives  of  each  component? 

- How  does  each  component  relate  to  the  overall  goals  of  the 
program? 

• Client/Consumer  Characteristics 

- What  are  the  characteristics  of  people  who  attend  the  program? 

- To  what  extent  is  the  program  reaching  the  people  it  was 
intended  to  reach? 
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• Program  Delivery 

- How  many  direct  contact  hours  (on  average)  does  each  client 
receive? 

- How  many  pamphlets  were  distributed? 

- How  many  clients  received  each  component  of  the  program? 

• Outcome  Evaluation 

- What  proportion  of  clients  complete  treatment? 

- What  proportion  of  clients  have  reduced  or  eliminated  substance 
abuse  following  treatment? 

- To  what  extent  have  clients  improved  their  level  of  self-esteem 
following  treatment? 

- To  what  extent  did  students  receiving  drug  education  increase 
their  knowledge  of  the  risks  of  drug  use? 

• Client/Consumer  Satisfaction 

- Which  parts  of  the  program  did  clients/consumers  find  most 
helpful?  least  helpful? 

- To  what  extent  did  clients  feel  their  needs  were  met? 

- Why  do  people  drop  out  of  the  program? 

- How  do  other  agencies  perceive  the  program  in  terms  of 
accessibility  and  quality? 

• Economic  Evaluation 

- What  is  the  average  cost  per  client  of  providing  the  program? 

- What  is  the  cost  per  client  of  the  day  program  compared  to  the 
outpatient  program? 

- What  proportion  of  program  costs  are  attributable  to  no-shows? 


Other  evaluation  terminology 

Readers  of  articles  or  books  on  program  evaluation  will  have  encountered  other 
evaluation  terms  than  the  ones  defined  here.  The  following  table  shows  how  the 
evaluation  categories  used  in  this  document  relate  to  other  commonly  used 
evaluation  terms. 


Evaluation  Terminology 


Level  of  Ev^iiiation 

Eyaluatidil  Terriis 

Description 

Needs  Assessment 

Collection  of  information  to  be  used  in 
developing  appropriate  interventions 
regarding  a specific  problem  or  goal. 

Logic  Model 

Program  Rationale 
Evaluability  Assessment 

Identification  of  hypothetical  causal  linkages 
between  what  is  done  by  the  program  and 
the  program’s  goals. 

The  Logic  Model  provides  a structured 
method  for  defining  the  rationale  of  the 
program. 

Client/Consumer 

Process  Evaluation 

Measuring  what  is  done  by  the  program  and 

Characteristics 

Implementation  Evaluation 
Formative  Evaluation 

to  whom. 

and 

Program  Delivery 

Input  Evaluation 

Although  identifying  client/consumer 
characteristics  and  monitoring  program 
delivery  are  generally  both  done  as  part  of 
process  evaluation,  this  casebook  considers 
the  two  aspects  separately. 

Outcome  Evaluation 

Effectiveness  Evaluation 
Summative  Evaluation 

Outcome  evaluation  involves  monitoring  the 
status  of  clients/  consumers  following  the 
intervention  compared  to  their  status  prior  to 
the  intervention  (e.g.,  decreased  alcohol 
consumed,  increased  knowledge  of  risks  of 
drug  taking). 

Effectiveness  or  Summative  Evaluation  uses 
outcome  information  with  appropriate 
control  conditions  to  assess  the  overall 
effectiveness  of  a program. 

Client/Consumer 

Satisfaction 

Client/Consumer  Feedback 

Obtaining  feedback  from  users  of  the 
program  regarding  their  perceptions  of  the 
program  (e.g.,  usefulness  of  the  program, 
pleasantness  of  program  environment,  etc.). 
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Level  of  Evaluation 

Related  Evaluation  Terms 

Description 

Economic  Evaluation 

Cost-Effectiveness 

Evaluation 

Cost-effectiveness  evaluation  combines  cost 
evaluation  and  effectiveness  evaluation.  It 
compares  alternative  program  options,  or 
program  vs.  "no  program"  options,  from  a 
particular  perspective  (e.g.,  the  perspective 
of  the  treatment  centre).  It  can  then  yield  an 
estimate  of  the  extra  cost  (or  cost  saving) 
incurred  in  providing  a specified  program 
per  unit  of  effectiveness  (e.g.,  per  additional 
smoker  who  quits  as  a result  of  the 
program). 

Cost-Benefit  Evaluation 

Cost-benefit  evaluation  attempts  to  determine 
whether  the  beneficial  consequences  of  a 
program  justify  its  cost;  i.e.,  to  assess  the 
net  monetary  benefit  gained  as  a result  of  a 
specified  program,  compared  to  an 
alternative  program  (or  a "no  program" 
condition).  For  cost-benefit  evaluations,  the 
consequences  of  alternative  programs  are 
valued  in  financial  terms,  and  the  difference 
in  these  values  is  compared  with  the 
difference  in  program  costs. 

Cost-Offset  Analysis 

The  term  "cost-offset  analysis"  is  sometimes 
used  to  describe  an  evaluation  that  considers 
a narrow  range  of  benefits  and  values  from  a 
very  limited  perspective  (e.g.,  an  evaluation 
to  determine  if  an  insurer’s  costs  in 
broadening  a policy’s  range  of  insured 
health  services  will  be  at  least  equalled  by 
the  insurer’s  savings  in  subsequent  periods). 

A step-by-step  approach  to  questions  involving  more  than  one  evaluation 
level 

Often  quality  enhancement  questions  involve  more  than  one  evaluation  level.  For 
example,  one  may  wish  to  know  whether  those  who  have  more  severe  alcohol 
problems  receive  more  services  than  those  who  have  less  severe  problems  (Client 
Characteristics  and  Program  Delivery).  Or  do  those  who  receive  more  services 
do  better  in  the  program  (Program  Delivery  and  Outcome  Evaluation)?  Or  do 
those  who  rate  the  drug  education  as  most  useful  also  show  the  greatest  increases 
in  knowledge  (Consumer  Satisfaction  and  Outcome  Evaluation)? 
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It  is  possible  to  break  down  each  of  these  questions  by  addressing  the  individual 
evaluation  levels  first,  and  then  using  this  information  to  consider  the  more 
complex  question.  For  example,  the  first  question  can  be  divided  into  three  steps. 
The  first  step  addresses  the  question:  "What  is  the  severity  of  alcohol  problems 
among  clients  who  enter  the  program?"  In  this  step,  each  client  who  enters  the 
program  during  a specific  period  (e.g.,  a year)  would  be  scored  according  to 
problem  severity.  To  use  a simple  scoring  method,  the  ratings  might  be  low 
severity,  medium  severity,  and  high  severity.  The  second  question  would  be: 
"How  many  hours  of  service  does  each  client  receive?"  The  program  might  use 
logs  to  record  all  time  spent  by  counsellors  with  each  client.  From  this  the  total 
number  of  hours  of  service  received  is  calculated  for  each  client.  With  these  two 
steps  completed,  the  information  is  now  available  to  address  the  original,  more 
complex,  question:  "Do  clients  who  have  more  severe  alcohol  problems  receive 
more  services  than  those  with  less  severe  problems?  To  answer  this  question,  the 
clients  can  be  grouped  into  three  categories:  low  severity,  moderate  severity,  and 
high  severity.  The  average  number  of  hours  of  service  for  each  group  can  then 
be  calculated.  Assume  the  following  results  were  obtained:  low  severity  - 4.2 
contact  hours,  moderate  severity  - 5.6  contact  hours,  and  high  severity  - 8.0 
contact  hours.  These  results  would  indicate  that,  indeed,  those  with  more  severe 
problems  receive  more  services.  This  information  might  help  the  program  to 
assign  appropriate  caseloads  or  to  investigate  other  more  efficient  means  of 
providing  extra  help  to  those  who  have  more  severe  problems. 

Each  case  example  is  broken  down  in  exactly  this  fashion.  Simple  questions  at 
one  level  of  evaluation  are  considered,  and  then,  for  most  examples,  more 
complex  questions  are  addressed  by  building  on  the  initial  questions. 


Questionnaires,  tables/graphs,  computer  software,  and  statistics 


Questionnaires.  Wherever  practical,  sample  questionnaires  are  included 
in  examples  that  use  questionnaires.  These  are  intended  to  show  the  manner  in 
which  questionnaire  data  are  used  to  address  quality  enhancement  questions. 

Tables  and  graphs.  Nearly  every  example  contains  tables  and  graphs. 
These  are  intended  to  demonstrate  typical  methods  for  summarizing  results. 


Computer  software.  While  the  focus  of  this  document  is  primarily  on 
using  evaluation  techniques  to  identify  and  address  quality  enhancement  questions, 
practical  approaches  to  analyzing  and  presenting  findings  (including  use  of 
computer  software)  were  included  to  help  facilitate  the  use  of  evaluation 
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techniques.  Specific  software  programs  for  IBM-compatible  PCs  are  described  and 
used  in  some  examples.  In  particular,  three  software  programs  are  described:  EPI 
INFO,  LOTUS  1-2-3,  and  Harvard  Graphics.  EPI  INFO  is  a "share-ware"  (i.e., 
can  be  copied  freely  for  use  by  others)  statistical  software  package  that  can 
produce  basic  statistics  for  people  who  do  not  have  extensive  knowledge  of 
statistics.  Of  the  software  investigated  for  use  as  an  example  for  this  casebook, 
EPI  INFO  seemed  to  be  the  most  accessible,  both  for  cost  (there  is  a $50.00 
charge  for  the  manual)  and  for  ease  of  use.  Some  analyses  and  graphs  are  shown 
using  LOTUS  1-2-3.  This  program  was  selected  for  use  in  the  casebook  because 
many  agencies  already  use  LOTUS  1-2-3.  Finally,  Harvard  Graphics  was  chosen 
as  a commercial  graphics  package  that  is  in  common  use.  It  should  be  stressed, 
however,  that  the  selection  of  these  programs  is  not  intended  as  a strategy  to 
promote  these  particular  products.  There  are  many  other  statistical  and  graphics 
software  programs  that  could  be  used  instead  of  the  ones  featured  in  this  casebook. 
However,  for  many  examples,  computers  would  be  useful,  if  not  essential,  for 
analyzing  the  data,  and  the  three  programs  described  in  detail  are  readily  available 
and  fairly  easy  to  use. 

The  examples  vary  in  level  of  difficulty  from  those  that  involve  minimal 
data  collection  where  the  results  can  be  tabulated  by  hand  to  fairly  sophisticated 
evaluation  research.  Most  examples  use  EPI  INFO  or  LOTUS  1-2-3  to  calculate 
statistics  and  LOTUS  1-2-3  and  Harvard  Graphics  to  generate  graphs.  A few 
more  complicated  examples  use  more  sophisticated  statistical  software  (SPSS). 
This  more  sophisticated  software  is  not  described  in  detail  within  the  casebook, 
since  background  training  for  using  this  software  goes  beyond  the  scope  of  the 
present  document. 

Statistics.  Most  examples  use  descriptive  statistics  (frequencies,  averages, 
percentages,  etc.).  In  a few  more  complex  examples,  inferential  statistics  are  also 
used.  Inferential  statistics  are  ones  that  assess  the  generalizability  of  findings  (e.g, 
t-test,  ANOVA,  Chi-square  test).  Findings  that  are  assessed  for  generalizability 
use  the  concept  of  "statistically  significant".  Since  the  present  casebook  is 
intended  to  demonstrate  some  basic  approaches  to  evaluation  and  not  intended  to 
teach  statistics,  statistical  significance  and  other  aspects  of  statistics  are  not 
explained  in  detail.  For  references  in  statistics,  see  the  annotated  bibliography. 
At  the  same  time,  the  casebook  is  likely  to  be  used  by  program  managers  and 
others  who  have  a range  of  background  education,  including  some  who  are  trained 
in  using  and  interpreting  inferential  statistics.  For  this  reason,  the  casebook 
includes  some  fairly  sophisticated  examples  (where  some  background  knowledge 
is  assumed),  as  well  as  some  fairly  easy  examples. 


m.  INDEX  OF  CASE  EXAMPLES  BY  RELEVANT  EVALUATION 
AND  QUALITY  ENHANCEMENT  CONCEPTS 


This  casebook  provides  concrete  (fictional)  examples  that  demonstrate  many  aspects  of  how 
addictions  programs  can  use  evaluation  techniques  (ranging  from  very  simple  questions  to  fairly 
large  scale  evaluations)  to  address  quality  enhancement  questions.  The  assumption  is  that  people 
can  learn  about  evaluation  by  reading  concrete  examples  of  evaluations  that  have  been  used  in 
programs  similar  to  the  particular  reader’s  program  of  concern.  For  example,  a program 
manager  of  an  addictions  treatment  program  might  first  examine  some  of  the  treatment 
examples.  The  first  concern  in  designing  the  casebook  was  that  there  be  a range  of  programs 
described  so  that  each  reader  can  find  an  example  using  a program  similar  to  the  program  with 
which  he  or  she  is  involved. 

Other  issues  that  help  frame  quality  enhancement  questions  include:  the  level  of  evaluation 
addressed  (e.g.,  client/consumer  characteristics,  program  delivery,  outcome,  economic);  the 
nature  of  the  client/consumer  population  (e.g.,  youth,  elderly,  women);  and  the  level  of 
knowledge  and  resources  available  for  addressing  the  quality  enhancement  questions.  For 
example,  a program  manager  might  be  considering  examining  consumer  satisfaction  and  might 
like  to  look  at  all  the  examples  that  look  at  this  issue.  A reader  interested  in  a specific  program 
for  women  might  want  to  look  at  examples  that  involve  women.  Finally,  a person  inexperienced 
in  evaluation  might  first  want  to  read  the  easy  examples  requiring  minimal  resources  to  get  a feel 
for  the  kinds  of  questions  with  which  he  or  she  might  be  able  to  begin  using  evaluation.  These 
issues  are  indexed  in  the  following  three  sections. 

The  final  section  provides  a comprehensive  index  to  a variety  of  evaluation  and  quality 
enhancement  practices  and  concepts  ranging  from  computer  software  to  confidentiality. 


Case  examples  by  level  of  evaluation 


The  examples  each  contain  evaluation/quality  enhancement  questions  for  several  levels  of 
evaluation.  The  following  shows  which  examples  address  each  level  of  evaluation. 


Needs  Assessment: 

Logic  Model: 

Client/Consumer  Characteristics: 

Program  Delivery: 

Outcome  Evaluation: 

Client/Consumer  Satisfaction: 

Economic  Evaluation: 

Client/Consumer  Characteristics  by  Program  Delivery: 
Client/Consumer  Characteristics  by  Outcome  Evaluation: 
Program  Delivery  by  Outcome  Evaluation: 

Client  Characteristics  by  Client/Consumer  Satisfaction: 
Program  Delivery  by  Client/Consumer  Satisfaction: 
Outcome  Evaluation  by  Client/Consumer  Satisfaction: 


Examples  El,  J1 
Examples  Al,  Bl, 
Examples  Cl,  D2, 
Examples  A2,  B2, 
Examples  A4,  B3, 
Examples  A3,  C2, 
Example  D6 
Examples  14 
Examples  L3 
Examples  D5,  J3 
Examples  C3,  H3 
Example  F4 


Dl, 

Gl, 

K1 

FI, 

G2, 

HI. 

, 11, 

J2,  LI 

D3, 

F2, 

12, 

K2, 

K3 

D4, 

G3, 

13, 

L2, 

L4 

F3, 

H2, 

K4 

Case  examples  by  client/consumer  population 
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In  addition  to  using  a range  of  addictions  programs  as  examples,  the  examples  include  a variety 
of  client/consumer  populations.  The  following  lists  the  consumer  populations  by  examples  that 
focus  on  these  populations. 


Youth: 

Older  Persons: 

Women: 

Parents  of  Adolescents: 

University  Students: 

Workplace: 

General  Population: 

Long-term  heavily  dependent  drinkers: 


Examples  A,  H 
Example  I 
Examples  G,  L 
Example  C 
Example  B 
Examples  A,  D,  F 
Examples  A,  E,  J,  K 
Example  J 


Case  examples  by  level  of  difficulty  and  resources  needed 

With  the  exception  of  the  Logic  Model,  each  part  of  the  examples  is  rated  for  the  level  of 
difficulty  and  level  of  resources  needed.  Parts  rated  "easy"  are  able  to  be  done  by  novice 
evaluators.  Those  rated  "moderate"  may  involve  more  knowledge  about  data  collection  and 
analysis,  or  may  involve  more  methodological,  measurement,  or  ethical  issues.  Parts  rated 
"difficult"  usually  require  some  knowledge  about  evaluation,  research  design,  and  statistics,  and 
would  not  be  appropriate  for  the  novice  evaluator  to  attempt. 

In  terms  of  resources,  a rating  of  "minimal  resources"  indicates  that  the  questions  can  be 
addressed  with  minimal  human  effort  and  no  special  equipment  or  materials.  Those  questions 
requiring  moderate  resources  are  more  likely  to  require  a computer  and  software,  and  they  may 
take  more  time  to  complete.  Those  questions  requiring  heavy  resources  would  not  be 
appropriate  for  most  agencies  who  do  not  have  in-house  evaluation  resources  or  funding 
available  for  supporting  the  evaluation. 

Easy/Minimal  Resources: 

Easy/Moderate  Resources: 

Easy/Heavy  Resources: 

Moderate/Minimal  Resources: 

Moderate/Moderate  Resources: 

Moderate/Heavy  Resources: 

Difficult/Minimal  Resources: 

Difficult/Moderate  Resources: 

Difficult/Heavy  Resources: 


Examples  A2,  A3,  Cl,  F2,  H2,  II,  14,  K4 
Examples  C2,  C3,  G2,  G3,  HI,  H3,  12,  13,  L2 


Examples  A4,  D3,  FI,  F4,  Jl,  J2,  K2,  K3,  LI 
Examples  B2,  D2,  D4,  D5,  F3 

Examples  L3 

Examples  B3,  D6,  El,  J3,  L4 
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Comprehensive  index:  Case  examples  by  evaluation  and  quality  enhancement 
practices  and  concepts 

EXAMPLES: 


alcohol  consumption  (measurement)  B3,  G2, 

annual  comparisons  D4,  G3, 

comparison  group  A4,  B3, 

computer  software 

customized  program  K2 

EPIINFO  A4,  FI, 

LOTUSl-2-3  A4,  C2, 

Harvard  Graphics  B2,  B3, 

SPSS  B2,  B3 

confidentiality  B3,  D3, 

control  group  A4,  B3, 

data  entry  (format)  B3,  C2, 

data  preparation  (coding)  B2,  B3, 

drug  use  (measurement)  G2,  G3, 

ethical  issues  B2,  B3, 

experimental  group  A4,  B3, 

external  factors  affecting  outcomes  A4,  B3, 

follow-up  B3,  D4, 

graphs 

area  graphs  J3,  K3 

bar  graphs  B2,  C2, 

line  graphs  A4,  B3, 

pie  charts  B2,  D2, 

mail-out  questionnaire  B2,  B3, 

measurement  issues  A4,  B2, 

methodological  issues  A4,  B2, 

J3,  K4, 

pre-test/post-test  B3,  D4, 

questionnaires 

sample  B2,  B3, 

standardized  B3,  El, 

referral  (measurement)  A4,  F4, 

release  of  information  form  F4 

response  rates  B2,  F3, 

response  bias  B2,  B3, 

return  rates  Jl,  K4 

scaling  Cl,  C2, 

self-selection  A4,  D5 

statistical  significance  B3,  H3, 

statistics 

ANOVA  (analysis  of  variance)  B3,  HI, 

averages  (also  called  means)  A4,  B2, 

K3,  K4, 

frequencies  G2,  G3, 

proportions  (also  called  percentages)  B2,  D2, 

14,  Jl,  J 


LI,  L3,  L4 

HI,  13,  J3,  K3,  L4 

D5 


HI,  H2,  H3,  Jl,  LI,  L2,  L4 

C3,  HI,  H2,  H3,  II,  12,  13,  14,  LI,  L2,  L4 

D2,  FI,  F2,  F3,  G2,  G3,  J2,  J3,  K2,  K3,  K4 

El,  F3,  H3,  Jl,  L4 
D5 

C3,  D4,  El,  HI,  H2,  H3,  LI,  L4 

C2,  C3,  D2,  El,  Jl 

13,  LI,  L3,  L4 

D3,  El,  F4,  Jl,  L4 

D5 

G3,  13 

D5,  E3,  E4,  G3,  13,  Jl,  J3,  L4 


FI,  F2,  F3,  G2,  H2,  14,  J2,  K4 
C3,  G3,  HI,  H3,  14,  L4 
FI,  G2,  II,  13,  K2,  K3,  LI,  L2 
Jl,  K4 

B3,  D6,  El,  F3,  HI,  H2,  13,  14,  LI,  L4 
B3,  D4,  D5,  El,  F3,  G2,  G3,  H2,  H3,  13,  14,  Jl, 
LI,  L4 
D5,  D6,  G2 

Cl,  H2,  Jl,  K4 
F3,  H2,  Jl,  K4 
HI,  J3,  K2,  K4,  L2 

Jl,  K4 
F3,  Jl 

C3,  F3,  H2,  Jl,  K4,  LI,  L4 
13,  J3,  K3,  L4 
H2,  H3,  L3,  L4 

B3,  C2,  C3,  F3,  F4,  HI,  H2,  H3, 12,  Jl,  J2,  J3,  K2, 
LI,  L2,  L3,  L4 
HI,  K4 

D3,  D4,  D5,  FI,  F2,  F4,  G2,  G3,  HI,  H2,  II,  13, 
2,  J3,  K2,  K3,  K4,  LI,  L2,  L3,  L4 


survey 

client/consumer 
community 
waiting  period 
waiting  list 


B2,  B3,  F3,  Jl 
El,  K4 
K2,  K3,  K4 
K2,  K3 


u 
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IV.  BASIC  FORMAT  USED  IN  THE  CASE  EXAMPLES 


Except  for  the  part  of  examples  where  a logic  model  is  presented,  each  part  of  the  examples  uses 
the  same  standard  format.  This  format  covers:  why  the  particular  questions  were  chosen,  the 
resources  needed,  how  the  information  was  collected  and  analyzed,  what  was  found  out  (i.e., 
the  results  of  the  evaluation),  how  the  findings  were  used,  and  the  methodological  measurement 
and  ethical  issues  relevant  to  the  example. 

The  one  part  of  some  examples  that  does  not  follow  this  format  is  the  logic  model.  The  logic 
model  is  a general  tool  used  to  understand  the  rationale  guiding  a program.  The  logic  model 
provides  an  evaluation  in  its  own  right  as  it  forces  the  program  to  undergo  scrutiny  and  clearly 
define  how  and  why  things  are  done  within  the  program.  In  addition,  the  logic  model  provides 
a tool  for  identifying  quality  enhancement  questions  that  are  likely  to  have  an  impact  on  program 
operations.  Finally,  the  logic  model  or  some  other  process  of  rationale  assessment  should 
always  be  completed  before  a large  scale  evaluation  is  undertaken. 


General  format  of  examples 

Each  case  example  consists  of  a cover  page  which  describes  the  program  and  identifies  the  levels 
of  evaluation  on  which  the  quality  enhancement  questions  are  based.  Each  level  or  combination 
of  levels  is  then  described  in  a separate  section. 

The  title  for  each  part  identifies  the  evaluation  level;  the  specific  quality  enhancement  questions 
are  listed  in  a box  following  the  title.  The  evaluation  is  then  detailed  in  the  format  outlined 
below. 


a.  Who  was  asking  the  question(s)  and  why  did  they  want  this  information? 

This  section  contains  the  rationale  for  asking  the  questions  and  describes  the  people  who  raised 
the  questions,  and  generally  how  the  information  will  be  used. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

This  section  includes  a description  of  the  materials,  equipment,  time,  and  human  resources 
required  to  collect,  analyze  and  interpret  the  information  to  address  the  identified  questions. 


c.  How  were  the  data  collected? 

This  section  describes  the  data  collection,  including  descriptions  of  questionnaires  and  other  data 
collection  methods. 
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d.  How  were  the  data  analyzed? 

This  section  includes  a description  of  the  organization  and  coding  of  the  data,  the  method  used 
to  analyze  the  data  (e.g.,  by  hand,  computer),  and  the  kinds  of  statistics  that  were  used  (e.g., 
statistics  such  as  averages,  frequencies)  to  summarize  the  results. 

e.  What  did  they  find  out? 

This  section  includes  a summary  of  the  results  of  the  data  analyses,  including  various  methods 
of  data  presentation  (e.g.,  tables,  graphs). 

/.  How  were  the  results  used? 

This  section  provides  some  possible  interpretations  of  the  results  and  describes  changes  that  were 
made  to  the  program  as  a result  of  the  findings. 


g.  Methodological,  measurement  and  ethical  issues. 

The  discussion  of  these  issues  is  located  in  a box  at  the  end  of  each  example.  Examples  were 
developed  to  highlight  most  of  the  methodological,  measurement  and  ethical  issues  likely  to 
occur  in  doing  evaluation.  The  comprehensive  index  in  Section  III  can  be  used  to  go  directly 
to  the  discussion  of  specific  issues. 


Logic  Model  Format 

Often  programs  have  several  components  or  activities.  For  example,  as  shown  in  Example  A, 
Drug  Awareness  Week  (DAW)  is  likely  to  have  several  different  components.  The  long-term 
outcome  or  goal  is  to  prevent  and  reduce  drug  and  alcohol  abuse.  In  order  to  find  out  if  this 
goal  was  achieved  it  would  be  necessary  to  collect  both  pre  and  post  DAW  drug  and  alcohol 
consumption  data  and  probably  some  follow-up  data  as  well.  This  data  collection  and  analysis 
would  be  time-consuming  and  expensive.  At  the  end  of  the  evaluation  it  still  might  not  be 
known  which  particular  components  were  effective  and  which  were  not. 

However,  it  is  possible  to  conduct  shorter-term  evaluations  of  each  component  of  the  program. 
To  use  this  approach,  it  is  often  helpful  to  draw  a logic  model  of  the  program.  A logic  model 
is  a diagram  or  flow  chart  that  shows  the  linkages  between  what  a program  does  and  what  it  is 
expected  to  achieve.  Each  component  can  then  be  followed  through  its  logical  stages  of 
implementation  and  outcome.  Because  each  component  can  be  viewed  separately,  it  can  be 
evaluated  separately.  Using  a logic  model,  a program  can  be  broken  down  into  its  components 
or  activities  to  illustrate  how  each  activity  leads  to  the  overall  goal.  The  following  is  the  format 
of  a basic  program  logic  model. 


MAIN  COMPONENTS 
(i.e.,  Activities/Resources) 


IMPLEMENTATION 

OBJECTIVES 

(e.g.,  to  provide,  to  give) 


OUTPUTS 

(e.g.,  countable  indicators  of  service 
delivery  and  characteristics  of  those 
served) 


SHORT-TERM  OUTCOME 
OBJECTIVES 

(e.g.,  to  increase,  to  decrease, 
to  maximize,  to  prevent) 


OUTCOME  INDICATORS 
(examples  of  measurable  indicators  that 
might  be  used  to  demonstrate  that 
outcome  objectives  have  been  achieved) 


LONG-TERM  OUTCOME 
OBJECTIVES 


LONG-TERM  OUTCOME 
INDICATORS 


□ 

□ 

□ 

□ 

□ 

1 

□ 

I 

1 

□ 

□ 

i 

□ 

i 

□ 

1 

□ 

1 

□ 

1 

1 

The  elements  of  the  model  are: 

MAIN  COMPONENTS  OR  ACTIVITIES:  To  identify  a program’s  main  components,  the 
evaluator  asks  "What  are  the  main  things  the  program  does  to  attain  its  goals?"  For  example, 
a component  of  Drug  Awareness  Week  might  be  ‘Campaign  Kit  for  Senior  Elementary  Schools.  ’ 

IMPLEMENTATION  OBJECTIVES:  These  objectives  state  what  the  program  seeks  to 
deliver  or  produce  and  the  targets  of  its  various  products  and  services.  At  least  one 
implementation  objective  should  be  developed  for  each  component.  Implementation  objectives 
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should  include  a strong  verb  and  a description  of  what  will  be  done.  For  example,  an 
implementation  objective  for  the  ‘Campaign  Kit’  component  might  be  ‘To  inform  school  staff 
about  resources.’ 

OUTPUTS:  These  are  the  countable  indicators  of  service  delivery  and  the  characteristics  of 
those  served.  For  the  ‘Campaign  Kit’  component,  the  outputs  would  be  the  number  of  kits 
distributed  and  the  number  of  schools  that  received  kits. 

SHORT-TERM  OUTCOME  OBJECTIVES:  These  short-term  or  intermediate  objectives  show 
how  the  program’s  main  components  are  expected  to  lead  to  the  attainment  of  goals  by  the 
recipient  of  the  program.  These  goals  are  expected  to  be  achieved  as  an  immediate  consequence 
of  the  program’s  activities.  A short-term  objective  of  the  ‘Campaign  Kit’  component  might  be 
‘To  increase  staff’s  use  of  ARF  and  other  resources.’ 

OUTCOME  INDICATORS:  These  are  the  indications  of  the  extent  to  which  the  outcome 
objectives  have  been  achieved.  An  outcome  indicator  of  the  ‘Campaign  Kit’  component  might 
be  ‘The  percent  of  school  staff  using  ARF.’ 

LONG-TERM  OUTCOME  OBJECTIVES:  This  is  often  referred  to  as  the  program’s  mission 
statement  or  long-term  goal.  The  long-term  goal  for  the  DAW  program  might  be  ‘To  reduce 
or  prevent  drug  abuse.  ’ 

LONG-TERM  OUTCOME  INDICATORS:  These  are  indicators  that  the  overall  goal  of  the 
program  has  been  achieved.  These  indicators  are  sometimes  difficult  to  measure  and  often 
difficult  to  attribute  solely  to  the  program  (outside  of  major  evaluation  research  projects).  It  is 
worthwhile  identifying  these  indicators,  however,  to  understand  what  the  program  really  hopes 
to  achieve.  For  example,  an  outcome  indicator  for  the  overall  goal  of  DAW  might  be  the  extent 
to  which  alcohol  consumption  has  gone  down  in  the  county  or  the  extent  to  which  drug  use  has 
been  reduced  in  schools. 


The  logic  model  provides  a clear  and  simple  overview  of  the  different  components  of  a program 
and  of  how  each  component  is  expected  to  contribute  to  the  overall  goal  of  the  program. 
Clearly,  such  a model  is  not  intended  to  capture  all  the  complexities  of  the  program  with  respect 
to  the  details  of  the  implementation  of  each  component  or  the  potential  interactions  of  the  various 
components.  It  is  the  simplicity  of  the  model  that  makes  it  a readily  useful  tool  for  approaching 
quality  enhancement  and  evaluation. 


V.  SUMMARY  OF  THE  CASE  EXAMPLES 


A wide  range  of  programs  have  been  chosen  to  demonstrate  the  use  of  evaluation  techniques  for 
addressing  quality  enhancement  questions.  Each  example  usually  contains  questions  at  several 
evaluation  levels  as  well  as  one  or  two  questions  using  a combination  of  evaluation  levels.  The 
case  examples  are  organized  into  three  categories:  health  promotion/prevention,  early 
intervention/secondary  prevention,  and  addictions  treatment. 


Health  Promotion/Primary  Prevention  Examples 

A.  Drug  Awareness  Week 

B.  University  Health  Promotion  Program 

C.  Parent  Education  Workshop 

D.  Workplace  Health  Promotion  Program 

E.  Community-Based  Needs  Assessment  for  Health  Promotion 

Early  Intervention/Secondary  Prevention  Examples 

F.  EAP  Counselling  Program 

G.  Community  Support  Program  for  Immigrant  Women 

H.  Prevention  Program  for  High-Risk  Youth 

Addictions  Treatment  Examples 

I.  Outreach  Treatment  Program  for  Older  Persons 

J.  Detox  Needs  Assessment  and  Evaluation 

K.  Assessment  and  Referral  Program 

L.  Women’s  Day  Program 
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VI.  CASE  EXAMPLES  A to  L 


si 


A.  Drug  Awareness  Week 


Program:  Drug  Awareness  Week  (DAW)  occurs  in  November  throughout  Canada.  It  is  a 
community  health  promotion  program  based  on  the  premise  that  awareness  is  the  key  to 
prevention.  Awareness  activities  occur  at  national,  provincial,  and  community  levels,  and  vary 
from  community  to  community  and  from  year  to  year.  Several  different  activities,  including  a 
drama  production  for  high  school  students  and  a media  campaign,  were  part  of  this  community’s 
DAW  program  during  the  year  of  the  evaluation. 

This  example  includes  quality  enhancement  questions  for  four  levels  of  evaluation: 


Part  1. 

Logic  Model 

Part  2. 

Program  Delivery 

Part  3. 

Consumer  Survey 

Part  4. 

Outcome  Evaluation. 
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PART  1.  LOGIC  MODEL 

LOGIC  MODEL  FOR  A DRUG  AWARENESS  WEEK  PROJECT 


MAIN 

COMI'ONKNIS 


imimj:mi:ntation 

OH.IKC'I  IVKS 


CAMPAIGN 

KIT  FOR 
SENIOR 
ELEMENTARY 
SCHOOLS 

DRAMA 

PRODCCTION 

FOR  GRADE  9 

I 

i 

To  inform  school 

To  inform  students 

staff  about  resources 

about  substance  abuse 

SUBSTANCE 
ABUSE  AND 

THE  WORKPLACE 
SEMINAR 

MEDIA 

CAMPAIGN 

t 

i 

To  provide  a 

To  inform  public 

seminar  to  inform 
workforce  referral 
agents  about 

Employee  Assistance 
Programs  (EAPs) 

about  the  event 

ou'ii'Lirs 


it  times  play  shown 
§ schools  seeing  play 
K students  seeing  play 


i 


ft  of  referral  agents 
attending  seminar 


it  of  public  service 
announcements  aired 
H articles  in  the 
press 


SHOR  r-TERM 

OUTCOME 

OB.IECTIVES 

To  increase  staffs 
use  of  ARE  and 
other  resources 

To  increase  students’ 
knowledge  almut  drug 
abuse 

To  increase  referral 
rate  to  EAPs 

To  increase  public’s 
awareness  of  the  event 

i 

i 

OUTCOME 

INDICATORS 

% of  school  staff 
using  ARF  resources 

% of  students  who 
showed  an  Increase  in 
knowledge  about  drug 
abuse 

% increase  in 
number  of  referrals 
to  EAP  programs 

% of  public  who 
report  being  aware  of 
DAW 

i 

1 

i 

i 

LONtMERM 

OUTCOME 

OB.IECTIVES 

To  reduce/prevent  drug  abuse  in  the  county 

LONG-TERM 

OU'ICOME 

INDICATORS 


i 


% decrease  in  per  capita  alcohol  consumption 
% decrease  in  drug  use  reported  by  high  school  students 

% decrease  in  alcohol  and  drug-related  problems,  such  as  liver  cirrhosis  and  drug-related 
emergency  department  admissions 
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Level  of  difficulty:  easy 

Level  of  resources  needed:  minimal 


Part  2.  Program  Delivery 


How  many  campai^  kits  were  distributed? 

How  many  students  saw  the  play? 

How  many  referral  agents  attended  the  EAP  seminar? 
How  many  pubUe  service  dnhbunc£mefUs  w^ 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 

Because  a lot  of  time  and  effort  goes  into  the  planning  and  implementation  of  DAW  activities, 
the  chair  of  the  DAW  Committee  felt  it  was  important  to  monitor  whether  the  programs  and 
materials  were  reaching  their  intended  audiences.  Before  any  outcome  evaluation  of  the  program 
could  be  done,  it  was  necessary  to  ensure  that  the  intended  audiences  received  the  program. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

One  person  on  the  committee  volunteered  to  be  the  "data  co-ordinator"  and  spent  a total  of 
fifteen  hours  co-ordinating,  tabulating,  and  analyzing  the  data  collected. 


c.  How  were  the  data  collected? 

The  person  involved  in  implementing  each  activity  was  responsible  for  recording  the  amount  of 
materials  distributed,  as  well  as  the  number  of  people  attending  the  plays  and  seminars.  All 
information  was  given  to  the  data  co-ordinator.  The  data  co-ordinator  also  contacted  the  media 
to  determine  the  number  of  public  service  announcements  aired  and  monitored  the  number  of 
articles  printed  in  area  newspapers  regarding  DAW.  Forms  for  describing  the  data  collected  for 
each  activity  are  shown  on  the  following  page. 
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(one  form  overall  for  recording  distribution  to  each  school) 

Campaign  Kits  Distributed 

School:  Date  distributed: 


(List  all  schools  provided  with  kits) 


Drama  Production  for  Grade  9 (one  form  for  each  school) 

Date:  School:  

Number  of  times  play  shown: 

Total  number  of  students  at  the  school  who  saw  the  play: 


Substance  Abuse  and  the  Workplace  Seminar 

Date:  Location  of  Seminar: 

(one  form  for  each  seminar  held) 

Number  of  people  invited  to  attend  the  seminar: 

Number  of  people  attending  the  seminar: 

Media  Campaign 

Number  of  public  service  announcements  aired  on  TV: 

Number  of  public  service  announcements  aired  on  radio: 

Number  of  articles  on  DAW  in  daily  newspapers: 

Number  of  articles  on  DAW  in  weekly  newspapers: 

(one  form  overall) 
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d.  How  were  the  data  analyzed? 

Totals  for  each  category  were  calculated  by  hand. 


e.  What  did  they  find  out? 

One  campaign  kit  was  distributed  to  each  of  the  40  elementary  schools  in  the  county. 

The  drama  production  for  Grade  9 students  was  shown  in  all  12  county  schools  that  had  Grade 
9 classes  and  approximately  1,050  students  saw  the  play. 

Of  the  50  referral  agents  who  were  invited  to  two  workplace  seminars,  21  attended. 

Finally,  nine  public  service  announcements  were  aired  (five  on  TV,  and  four  on  radio)  during 
the  week  and  six  articles  appeared  in  area  newspapers  (five  in  the  daily  newspaper  and  one  in 
a weekly  neighbourhood  paper). 


/.  How  were  the  results  used? 

The  information  collected  helped  the  DAW  Committee  to  determine  that  the  program  was 
reaching  its  intended  audience  in  the  schools  and  in  the  community.  There  was  some  concern, 
however,  about  the  low  attendance  at  the  workplace  seminar.  One  person  on  the  DAW 
Committee  has  volunteered  to  contact  the  29  referral  agents  who  did  not  attend  the  seminar  in 
order  to  receive  feedback  about  why  they  did  not  attend.  The  results  of  this  survey  are 
presented  in  Part  3 of  this  example. 

Since  most  components  of  DAW  week  appeared  to  have  been  implemented  appropriately,  a 
decision  was  made  to  try  to  monitor  some  outcome  indicators  during  the  next  DAW  as  well  as 
continuing  to  monitor  program  delivery.  Monitoring  outcome  indicators  is  described  in  Part  4 
of  this  example. 
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Level  of  difficulty:  easy 

Level  of  resources  needed:  minimal 


Part  3.  Consumer  Survey 


What  were  the  main  reasons  for  poor  attendance  at  the  workplace 
seminars? 


a.  Who  was  asking  the  question  and  why  did  they  want  this  information? 

The  DAW  Committee  was  concerned  that  less  than  50%  of  the  referral  agents  who  were  invited 
to  attend  the  workplace  seminars  actually  attended.  The  committee  recognized  that  there  had 
been  little  advance  notice  of  the  seminars  and  that  the  times  during  which  they  had  been 
scheduled  were  not  ideal.  Therefore,  they  wanted  to  determine  whether  these  problems  prevented 
people  from  attending  the  seminars.  Also,  they  wanted  to  identify  any  additional  reasons  why 
people  did  not  attend. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

One  volunteer  from  the  DAW  committee  spent  a total  of  15  hours  contacting  those  referral 
agents  who  did  not  attend  the  seminar  by  telephone,  categorizing  the  responses,  and  tabulating 
the  reasons  for  non-attendance. 


c.  How  were  the  data  collected? 

A volunteer  from  the  DAW  committee  contacted  each  of  the  29  referral  agents  who  did  not 
attend  the  seminar  and  asked  the  following  two  questions: 

1 . Were  there  barriers  which  made  it  difficult  to  attend  the  seminars? 

2.  Do  you  have  any  suggestions  for  making  future  seminars  more  accessible? 
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d.  How  were  the  data  analyzed? 
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The  volunteer  tabulated  the  responses  and  then  divided  them  into  the  following  categories  of 
reasons  given  for  not  attending  the  seminar: 

1.  scheduled  seminar  times  were  not  convenient 
2 not  enough  advance  notice 

3.  seminar  topic  of  low  priority  to  company 

4.  other. 


e.  What  did  they  find  out? 

The  volunteer  was  able  to  reach  all  of  the  29  referral  agents  who  did  not  attend  the  seminar. 
Each  referral  agent  responded  to  the  questions  asked.  The  number  of  referral  agents  who  were 
in  each  response  category  is  shown  in  Table  1.  Note  that  some  agents  gave  more  than  one 
response;  therefore,  the  total  number  of  responses  is  greater  than  29.  Some  examples  of 
comments  are  also  included  in  the  table. 
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Table  1.  Number  of  Referral  Agents  Indicating  Each  Category  of  Reason  for  Not 
Attending  the  Workplace  Seminar  (n=29) 


Reasons  for  Hot  attending 

Number  of  agents 
indicating  each 
reason 

Examples  of  types  of  comments 
made 

scheduled  seminar  times  were 
not  convenient 

20  (69%) 

• seminars  were  too  long  (2  hrs.) 

• could  try  lunch  hour  seminar 

• there  was  a conflict  with  other 
commitments 

• prefer  mid-week 

not  enough  advance  notice 

23  (79%) 

• staff  usually  schedule  themselves  three 
weeks  in  advance 

• all  staff  had  prior  commitments 

• need  at  least  1 month’s  notice 

seminar  topic  not  a high 
priority  within  agency 

7 (24%) 

• not  enough  referrals  made  to  EAP  to 
warrant  taking  time  to  attend 

• another  seminar  at  the  same  scheduled 
time  was  more  relevant  to  me 

other 

6 (21%) 

• forgot 

• was  ill 

• last  minute  change  in  schedule 

/.  How  were  the  results  used? 

Given  that  a majority  of  the  referral  agents  indicated  that  they  required  more  advance  notice  to 
have  staff  available  to  attend,  and  given  that  scheduling  of  the  seminars  was  an  issue,  the  DAW 
committee  has  decided  to  schedule  one  noon-hour  seminar  as  well  as  a longer  (two  hour)  evening 
seminar  next  year.  They  also  plan  to  set  a target  for  distribution  of  information  about  the 
seminars  at  least  four  weeks  prior  to  the  scheduled  seminar  dates.  With  regards  to  the  schedule 
conflict  with  the  other  seminar,  the  committee  felt  that  although  they  could  not  be  assured  that 
this  would  not  happen  again,  the  planned  changes  would  increase  attendance  at  the  workplace 
seminars  next  year. 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  moderate 


Part  4.  Outcome  Evaluation 


Did  knowledge  about  dmg  Wuse  increase  (unong  Grade  9 students  wh^ 
saw  the  play? 

Did  the  nuntber  of  rtferrats  to  EAP  program  increase  ? 


a.  Who  was  asking  the  question  and  why  did  they  want  this  information? 

The  DAW  Committee  was  interested  in  examining  outcomes  because  they  wanted  to  know  if 
their  efforts  appeared  to  be  making  a difference.  In  particular,  the  drama  production  and  the 
workplace  seminar  were  quite  labour  intensive  and  the  committee  wanted  to  be  sure  they  were 
worth  the  effort  if  they  were  to  be  continued  next  year. 

In  addition,  the  Board  of  Education  for  the  county  and  the  community  members  of  the  workplace 
seminar  committee  who  worked  with  the  DAW  Committee  planning  the  interventions  were 
concerned  that  their  time  had  been  well  spent. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

One  committee  member  worked  a total  of  five  days  collecting  and  tallying  information  regarding 
the  effects  of  the  workplace  seminar.  Two  committee  members  worked  five  days  each 
scheduling  and  testing  students  and  analyzing  data  for  the  evaluation  of  the  play  for  Grade  9 
students. 


c.  How  were  the  data  collected? 

The  play.  Because  people  wanted  to  know  whether  students’  knowledge  had  increased  as  a 
result  of  seeing  the  play,  it  was  decided  to  test  students’  knowledge  before  and  after  the  play. 
However,  to  be  sure  that  it  was  the  play  that  was  causing  any  change  that  might  occur,  it  was 
also  decided  to  use  a control  group,  that  is,  to  test  some  Grade  9 students  who  did  not  see  the 
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play.  In  order  to  do  this  it  was  necessary  to  make  arrangements  with  some  schools  in  an 
adjacent  county  where  a school  drug  and  alcohol  education  program  was  not  part  of  DAW.  One 
week  prior  to  the  week  of  the  performance  committee  volunteers  gave  a knowledge  test  to 
students  from  five  classes  who  were  to  see  the  play  and  students  from  five  classes  in  the  control 
county.  Two  weeks  after  the  students  saw  the  play  both  groups  of  students  were  retested.  The 
knowledge  questionnaire  used  is  reproduced  at  the  end  of  this  example.  A 10  digit  code 
comprised  of  the  respondent’s  date  of  birth  (mm/dd/yy)  and  the  last  four  digits  of  their  telephone 
number  was  used  to  match  before  and  after  responses. 

The  workplace  seminar.  Six  months  after  the  EAP  seminar,  a DAW  Committee  member 
telephoned  all  the  workplace  referral  agents  who  had  been  invited  to  attend  the  seminar.  Each 
agent  was  asked  to  examine  his  or  her  records  and  assess  how  many  referrals  had  been  made 
to  EAP  programs  in  the  months  prior  to  DAW  and  how  many  had  been  made  in  the  six  months 
following  DAW. 


d.  How  were  the  data  analyzed? 

The  play.  The  test  data  was  entered  into  the  computer  and  was  analyzed  using  EPI  INFO. 
Before  and  after  knowledge  scores  were  compared  for  the  group  who  saw  the  play  and  the  group 
that  did  not.  The  change  in  scores  were  computed  and  the  average  change  was  compared  for 
the  students  who  saw  the  play  and  those  who  did  not. 

The  workplace  seminar.  The  average  number  of  referrals  to  EAP  by  each  referral  agent  was 
calculated  by  hand  for  the  six  months  prior  to  and  following  DAW  for  those  who  attended  the 
seminar  and  for  those  who  did  not.  The  amount  of  change  in  the  number  of  referrals  was 
computed  and  compared  for  those  who  attended  the  seminar  and  those  who  did  not. 

Line  graphs  comparing  the  before  and  after  results  for  each  of  the  two  components  were 
prepared  using  Lotus  1-2-3  (see  Appendix  B). 


e.  What  did  they  find  out? 

The  play.  As  shown  in  Table  1 , students  who  saw  the  play  and  those  who  did  not  scored  about 
the  same  on  the  knowledge  test  before  the  play  was  shown.  Those  students  who  saw  the  play 
increased  their  knowledge  scores  substantially,  while  those  who  did  not  see  the  play  increased 
their  knowledge  only  slightly.  There  was  a significant  difference  in  the  change  in  knowledge 
scores  between  the  two  groups.  Figure  1 shows  these  results  in  a line  graph  which  was 
generated  using  LOTUS  1-2-3. 
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Table  1.  Average  Scores  on  the  Knowledge  Test  Before  and  After  the  Play  (Students  Who 
Saw  the  Play  Compared  With  Those  Who  Did  Not) 


Experimental 

n=160 

Control 

n=164 

Average  scores  on  knowledge  test  (out  of  10): 

before  (F=1.8,  df=322,  n.s.) 

4.1 

4.4 

after  (F  = 174.7,  df=322,  p<.001) 

8.8 

6.2 

change  (F=  177.9,  df=322,  p<.001  ) 

4.7 

1.8 

Figure  1.  Average  Scores  on  the  Knowledge  Test  Before  and  After  the  Play  was  Shown 
(Students  Who  Saw  the  Play  Compared  With  Those  Who  Did  Not) 
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The  workplace  seminar:  As  shown  in  Table  2,  prior  to  the  workplace  seminar,  agents  who 
attended  the  seminar  did  not  differ  significantly  from  those  who  did  not  attend  on  the  number 
of  referrals  to  EAPs.  After  the  seminar,  the  rate  of  referrals  by  those  who  did  not  attend 
decreased  slightly,  while  the  rate  of  referrals  more  than  doubled  for  those  who  attended  the 
seminar  (see  Table  2).  There  was  a significant  difference  in  the  change  in  the  number  of 
referrals  between  the  two  groups.  Figure  2 further  illustrates  these  results  using  a line  graph. 


Table  2.  Average  Number  of  Referrals  Made  to  EAP’s  Before  and  After  the  Workplace 
Seminar  (Referral  Agents  Who  Attended  the  Seminar  Compared  With  Those 
Who  Did  Not) 


Attended 

n=21 

Did  Not  Attend 
n=29 

Average  number  of  referrals  to  EAP: 

during  six  months  before  seminar 
(F=1.8,  df=49,  n.s.) 

1.2 

1.6 

during  six  months  after  seminar 
(F  = 23,8,  df=49,  p<.001) 

2.6 

1.1 

change  in  number  of  referrals 
(F=30.3,  df=49,  p<.001 

+ 1.4 

-0.6 

Figure  2.  Average  Number  of  Referrals  Made  to  EAP’s  Before  and  After  the  Workplace  Seminar 
(Referral  Agents  Who  Attended  the  Seminar  Compared  With  Those  Who  Did  Not) 


Before  After 

j Attended  the  seminar  4 Did  not  attend  seminar 
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/.  How  were  the  results  used? 


The  positive  results  from  the  play  performance  and  the  workplace  seminar  indicated  that  these 
are  useful  components  to  continue  as  part  of  DAW. 


Methodological,  measurement  and  ethical  issues: 

Methodological.  It  is  important  to  make  sure  that  students  in  the  classes  who  did  not  see  the 
play  were  not  given  ’catch  up’  lessons  by  their  teachers  because  the  teachers  felt  their  students 
were  missing  out.  If  those  students  were  taught  lessons  containing  similar  information  to  that  in 
the  play,  students  who  saw  the  play  would  not  be  expected  to  score  any  higher  than  students  who 
did  not  see  it. 

Referral  agents  who  attended  the  seminar  may  have  been  more  interested  in  increasing  their 
referrals  to  EAP  than  those  who  did  not  attend.  Therefore,  the  increase  in  referrals  for  those 
attending  the  seminar  may  be  partly  a reflection  of  self-selection  rather  than  entirely  due  to  the 
seminar  itself. 

Measurement.  When  knowledge  is  being  evaluated,  it  is  important  that  the  testing  instrument 
focus  on  questions  that  reflect  information  that  was  provided  as  part  of  the  intervention:  in  this 
case,  the  play. 


Drug  Awareness  Week 


32 


Grade  9 Knowledge  Test 

The  following  statements  are  either  True  or  False.  CIRCLE  the  alternative  which  you  believe 
to  be  correct.  If  you  don’t  know  the  answer,  do  not  circle  either  of  the  alternatives. 


1.  A regular  bottle  of  beer  has  as  much  alcohol  in  it  as  one  shot  (1  Va  oz.  or  43  mL) 

of  whisky.  T F 

2.  Cola  drinks  do  not  contain  caffeine.  T F 

3.  Two  or  three  marijuana  cigarettes  a day  may  carry  the  same  risk  of  lung  damage 

as  a pack  of  tobacco  cigarettes.  T F 

4.  Alcohol  can  help  cure  a cold.  T F 

5.  You  cannot  be  an  alcoholic  if  you  only  drink  beer.  T F 

6.  There  is  no  possibility  of  physical  or  psychological  addiction  to  marijuana.  T F 

7.  Long-term  cocaine  use  can  cause  nose  bleeds.  T F 

8.  Alcohol  leaves  the  body  for  the  average  adult  male  at  the  rate  of  one  drink  per  hour.  T F 

9.  Each  tobacco  cigarette  smoked  shortens  your  life  by  about  10  minutes.  T F 

10.  Drinking  coffee  can  sober  you  up.  T F 
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B.  University  Health  Promotion  Program 


Program:  In  collaboration  with  researchers  and  community  representatives,  University  A 
developed  a health  promotion  program  intended  primarily  for  first  year  students.  The  program 
stressed  health  promotion  and  responsible  behaviour  toward  alcohol  use.  First  year  students 
were  chosen  to  be  the  primary  target  for  several  reasons.  First,  these  students  are  entering  an 
environment  with  which  they  are  unfamiliar,  and,  for  many  students,  this  will  be  their  first 
experience  living  away  from  home.  Second,  most  students  reach  the  legal  drinking  age  while 
at  university  and  are  then  able  to  purchase  and  drink  alcohol  legally.  Third,  there  is  a certain 
mystique  and  stereotype  about  university  students  and  alcohol  use  which  students,  upon  entering 
the  system,  seem  determined  to  perpetuate.  Fourth,  the  patterns  and  behaviours  which  students 
adopt  in  their  first  year  of  university  may  set  the  trend  for  their  future  experiences. 

The  program  was  made  up  of  two  independent,  yet  related  components  - an 
educational/persuasion  component  and  a policy  component  - both  of  which  stressed  the  health 
promotion  perspective.  The  basic  message  of  the  education/persuasion  campaign  stressed  the 
ideas  of  "personal  moderation"  and  the  personal  responsibility  which  students  must  take  for  their 
own  behaviour.  The  messages  were  promoted  by  the  use  of  a number  of  channels  including  a 
core  pamphlet  sent  to  all  incoming  first  year  students,  theme  flyers,  awareness  posters, 
advertisements  in  the  student  newspaper,  presentations  to  students  in  residence,  and  messages 
in  licensed  outlets  on  campus.  Four  alcohol-related  and  potentially  damaging  behaviours  were 
emphasized:  the  health  effects  of  alcohol  consumption;  drinking  and  driving;  drinking  to 
drunkenness;  and  drinking  before  undertaking  academic  activities. 

The  policy  component  required  the  development  and  implementation  of  policies  which  were  to 
govern  and  regulate  the  sale  and  use  of  alcohol  on  campus  for  all  sectors  of  the  university. 
Both  components  were  implemented  prior  to  the  beginning  of  classes  in  September  and  were  in 
effect  throughout  the  academic  year. 

This  example  includes  quality  enhancement  questions  for  three  levels  of  evaluation: 


Part  1. 

Logic  Model 

Part  2. 

Program  Delivery 

Part  3. 

Outcome  Evaluation. 
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Part  1.  Logic  Model 

Logic  Model  for  University  Health  Promotion  Program 


PAMPHLETS 
MAILED  TO  FIRST 
YEAR  STUDENTS 

MEDIA 

CAMPAIGN 

ALCOHOL 
MANAGEMENT 
POLICY  ON  CAMPUS 

1 

1 

To  inform 
incoming  students 
about  low-risk 
drinking  behaviour 

To  inform  student 
body  about  low-risk 
drinking  behaviour 
and  reinforce 
messages  in  booklets 

To  implement  a series  of 
policies  in  campus  bars 
including  differential 
pricing  of  beer  based  on 
alcohol  content,  non- 
alcoholic alternatives  at 
reasonable  prices,  etc. 

i 

; 

1 

ff  students 
receiving  pamphlets 
# students  aware 
of  message 

H students  aware 
of  campaign 
# students  aware 
of  message 

# policy  options  adopted 

# students  aware  of  policy 
options 

\ 

/ 

1 

To  increase 
knowledge  about 
and  attitudes  in 
favour  of  low-risk 
drinking  practices 

To  increase  approval  and 
adoption  of  tow  risk 
drinking  practices 

I 

# students  who 
increased  knowledge 
about  and  attitudes 
in  favour  of  low-risk 
drinking  practices 

# students  who  approved 
and  adopted  low  risk 
drinking  practices 

/ 

Maintenance  of  responsible  drinking 
behaviour 


I 

% increase  in  rate  of  students  at  the 
university  who  drank  responsibly 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  heavy 


Part  2.  Program  Delivery 


IMd first  year  students  rec^^  and  were  they  aware 

(^the  message  in  the 


Were  students  aware  of  the  media  campaign  and  aware  of  its  messages? 

How  many  alcohol  management  policies  were  implemented^  and  to  what 
extentM^re  students  aware  0^^^^ 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 

The  team  of  university,  community  and  research  representatives  were  aware  that  in  order 
for  the  program  to  have  any  impact,  students  needed  to  receive  the  program  information  and  be 
aware  of  its  content.  Therefore,  a survey  of  first  year  students  was  conducted  following  the 
intervention  to  determine  whether  the  students  who  received  the  intervention  were  aware  of  the 
program  and  its  various  components.  Also,  the  types  of  policies  implemented  by  the  university 
community  were  monitored. 


h.  What  resources  were  needed  to  collect  and  interpret  the  information? 

Human  resources  were  required  to  package  and  send  the  surveys  to  students  randomly  selected 
by  the  universities,  to  monitor  the  media  campaign,  and  to  enter  the  data.  Printing  and  paper 
costs  were  also  incurred  for  the  survey.  Additional  resources  were  required  for  postage  to  mail 
the  questionnaires  to  students  and  for  return  postage  for  questionnaires  returned  by  students. 
This  component  of  the  evaluation  required  two  people  working  five  full  days  to  package  and 
send  out  the  questionnaires.  Monitoring  was  done  by  the  research  assistant  at  periodic  intervals 
and  data  entry  was  contracted  out  at  an  approximate  total  cost  of  $1,000.  The  research  assistant 
required  several  months  to  check  the  data  for  inconsistencies  and  other  errors  and  to  analyze  the 
findings.  The  data  analyses  required  a computer  and  statistical  software. 
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c.  How  were  the  data  collected? 

Seven  months  after  the  beginning  of  the  school  year,  questionnaires  were  sent  to  1,500  first  year 
students  who  had  been  randomly  selected  from  all  first  year  students  at  the  university.  A self- 
addressed  stamped  envelope  was  enclosed  for  return  of  the  completed  questionnaires.  The 
questionnaires  were  to  be  completed  anonymously,  asking  for  sex,  date  of  birth  and  the  last  four 
digits  of  their  parents’  telephone  number  as  the  only  identifiers. 

On  the  questionnaire,  students  were  asked  a series  of  questions  about  their  awareness  and 
understanding  of  the  various  components  of  the  program  during  the  past  school  year  (see  the 
questionnaire  provided  at  the  end  of  this  part  of  the  example). 


d.  How  were  the  data  analyzed? 

All  information  on  the  returned  surveys  was  coded  and  entered  for  analysis  by  computer  using 
SPSS  (Statistical  Package  for  Social  Sciences)  software.  For  these  analyses,  coding  the 
questionnaires  involved  assigning  numerical  values  for  all  questionnaire  items  that  were  to  be 
tabulated.  For  example,  for  questions  1(a),  1(b),  and  2(a),  "yes"  was  coded  "1"  and  "no"  was 
coded  "2".  For  question  2(b),  "reading  before  arriving  on  campus"  was  coded  "1",  "reading 
after  arriving"  was  coded  "2",  and  "not  reading"  was  coded  "3".  The  ratings  for  each  of  the 
three  scales  in  question  2(c)  were  scored  1 to  5 and  averaged. 

Graphs  were  prepared  using  Harvard  Graphics  (see  Appendix  C)  to  illustrate  the  results. 


e.  What  did  they  find  out? 

The  pamphlet.  Approximately  40%  of  first  year  students  responded  to  the  mailing.  Seventy- 
seven  percent  of  respondents  reported  that  they  had  received  the  core  pamphlet  at  home,  and  of 
these,  62%  said  that  they  read  the  pamphlet  before  arriving  on  campus  and  another  10%  read 
it  after  arriving  on  campus.  Figure  1 illustrates  these  results  in  a pie  graph  generated  using 
Harvard  Graphics.  Respondents  who  read  the  booklet  generally  found  it  interesting,  informative 
and  useful.  The  average  ratings  (out  of  3)  are  shown  in  a bar  graph  in  Figure  2.  Almost  84% 
of  the  respondents  were  aware  of  the  program’s  motto,  and  of  these,  90%  reported  that  the 
motto  was  meaningful  to  them. 

The  media  campaign.  The  media  campaign  covered  four  main  topics.  Eighty-one  percent  of 
respondents  reported  seeing  or  hearing  educational  messages  for  at  least  one  topic.  The  most 
frequently  remembered  topic  was  drinking  and  driving  (78%),  followed  by  drinking  and  learning 
(65%),  drinking  to  drunkenness  (59%),  and  drinking  and  physical  health  (43%)  (see  Figure  3). 
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Alcohol  management  policy.  Eleven  policies  supporting  lower  risk  drinking  practices  were 
implemented  on  the  university  campus  (e.g. , encouraging  lighter  beer  or  non-alcohol  alternatives 
in  campus  bars). 

Of  survey  respondents  who  reported  that  they  had  attended  campus  bars  and/or  special  pub 
nights,  37%  had  noticed  the  "Drink  Light"  table  signs,  57%  were  aware  that  light  beer  was 
cheaper  and  89%  said  they  approved  of  lower  pricing  for  lower  alcohol  beer.  However,  only 
20%  of  those  who  attended  campus  bars  and/or  pub  nights  reported  that  the  lower  price  for  low 
alcohol  beer  influenced  them  to  purchase  low  alcohol  beer.  These  were  the  only  policies 
implemented  that  were  visible  to  patrons,  and  were  the  only  ones  included  in  the  survey  (see 
Figure  4). 


Figure  1 . Percent  of  Students  Who  Read  the  Pamphlet  Before 
Arriving  on  Campus,  After  Arriving  on  Campus,  or  Not  at  All 


Read  before  arriving 

62% 
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Figure  2.  Average  Ratings  of  Pamphlets  on  Level  of  Interest, 
Informativeness,  and  Usefulness 

Average  Rating  (out  of  5) 

Scale  0 1 2 3 4 5 


Level  of  Interest 


Informativeness 


Usefulness 
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Figure  3.  Percent  of  Respondents  Aware  of  Media  Campaign 

Messages 

Awareness  of:  ^0%  40%  60%  80%  100% 

At  least  one  message 

Drinking/Driving 

Drinking/Learning 

Drinking/Drunkeness 

Drinking/Phys.  health 
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Figure  4.  Percent  of  Students  Reporting  Awareness  and  Approvai 
of  Poiicies  Impiemented  by  Bars 


% of  students 

Poiicies  Impiemented  o%  20%  40%  60%  80%  100% 


Aware/drink  iight  sign 
Aware/d  iff.  pricing 
Approve/diff.  pricing 
Bought  iow  aic.  beer 


/.  How  were  the  results  used? 

The  results  indicated  that  the  program  was  successful  in  reaching  the  target  audience  of  first  year 
students.  It  also  appeared  that  the  majority  of  those  who  responded  to  the  survey  remembered 
the  content  of  the  pamphlets  and  the  media  messages.  In  addition,  reaction  to  the  messages  and 
the  policies  implemented  in  the  campus  bars,  such  as  differential  pricing  based  on  alcohol 
content,  were  positively  received  by  respondents. 

This  information  was  used  to  promote  discussion  about  implementing  the  program  on  a regular 
basis  within  the  university  setting. 
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Methodological,  measurement,  and  ethical  issues: 

Methodological.  The  questionnaire  used  in  this  part  of  the  study  was  actually  part  of  a 
larger  follow-up  questionnaire  administered  as  part  of  the  outcome  evaluation. 

Another  methodological  issue  is  the  possible  impact  of  the  40%  response  rate.  It  is 
possible  that  those  who  responded  were  more  likely  to  remember  receiving  the  pamphlets 
and  more  likely  to  respond  positively  to  responsible  drinking  policies.  This  potential  bias 
needed  to  be  taken  into  consideration  in  interpreting  the  results. 

Measurement.  While  it  was  relatively  easy  to  monitor  the  educational  aspects  of  the 
intervention  (i.e.,  whether  the  students  read  the  pamphlets),  it  was  more  difficult  to  monitor 
the  implementation  of  alcohol  management  policies.  It  was  possible  to  determine  whether 
the  alcohol  advisory  committee  sanctioned  a particular  policy,  but  it  was  generally  difficult 
to  ascertain  whether  the  bar  managers  and  serving  personnel  implemented  the  policy  as 
required.  In  fact,  it  became  apparent  during  the  evaluation  that  a number  of  the  policies 
which  had  been  adopted  were  viewed  as  unworkable,  or  were  unpopular  with  managers, 
and  violations  of  these  policies  were  routinely  ignored.  For  example,  ensuring  that  food 
was  readily  available  on  all  occasions  when  alcohol  was  sold  was  deemed  appropriate  but 
difficult  to  enforce  in  certain  types  of  settings. 

Ethical.  Because  the  implementation  of  approved  university  policies  was  monitored,  the 
potential  for  finding  bar  staff  not  following  through  on  these  policies  was  discussed. 
Would  these  violations  be  reported  to  the  administration?  It  was  decided  that  enforcing 
policies  was  the  responsibility  of  the  university  and  that  it  would  be  inappropriate  for  the 
program  evaluators  to  report  specific  deviations  from  the  policies. 
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4. (a)  Have  you  been  in  any  campus  bars  or  attended  any  special  event 
pub  nights  on  campus  since  September? 

YES  NO 
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Level  of  difficulty:  difficult 
Level  of  resources  needed:  heavy 


Part  3.  Outcome  Evaluation 


Did  students  exposed  less  than  students  who  were  not 

exposed  to  the  program? 

Did  attitudes  improve  among 

students? 

Did  knowledge  ahout  alcohol  use  among  students? 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 


The  university  and  other  collaborators  on  the  project  wanted  to  determine  the  effectiveness  of 
the  program.  If  the  program  could  be  shown  to  be  effective  in  actually  improving  drinking 
attitudes  and  behaviours,  it  would  be  worthwhile  to  continue  it.  In  addition,  this  would  provide 
a model  program  for  other  universities  to  adopt.  To  address  the  effectiveness  of  the  program, 
questionnaires  were  sent  to  students  at  University  B where  the  program  had  not  been 
implemented  (i.e.,  "control"  university),  as  well  as  to  students  at  University  A where  the 
program  had  been  implemented  (i.e.,  "experimental"  university). 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

The  study  involved  sending  two  sets  of  questionnaires  to  3,000  students.  In  addition  to  the  costs 
of  the  questionnaires,  human  resources  were  needed  to  process  the  returned  questionnaires  and 
to  enter  and  analyze  the  data.  Access  to  a computer  and  a statistical  software  package  was 
required.  Because  the  dataset  was  large,  data  management  and  analysis  were  quite  time- 
consuming.  For  example,  several  weeks  were  needed  to  clean  the  data  set  (e.g.,  correct  data 
entry  errors)  and  prepare  the  data  for  analyses  (e.g.,  compute  variables  from  questionnaire 
items).  Overall,  the  budget  for  the  evaluation  component  of  the  project  was  about  $30,000  and 
required  a half-time  research  assistant  and  about  20%  of  a researcher’s  time  over  a one-year 
period. 
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c.  How  were  the  data  collected? 

A set  of  questionnaires  was  mailed  at  the  end  of  August  (before  students  arrived  on  campus)  to 
3,000  first  year  students  who  had  been  randomly  selected  from  the  two  universities.  The 
questionnaire  asked  a number  of  questions  about  knowledge  about  alcohol  and  drinking  practices 
and  attitudes.  The  questionnaires  were  completed  anonymously  with  sex,  date  of  birth  and  the 
last  four  digits  of  their  parents’  telephone  number  as  the  only  identifiers. 

Eight  months  later,  after  the  program  had  been  implemented  at  University  A,  a second 
questionnaire  was  mailed  to  the  same  3,000  students.  The  follow-up  questionnaire  was  the  same 
as  the  initial  questionnaire  with  slight  changes  to  reflect  time  on  campus.  The  questionnaires 
mailed  to  students  at  the  experimental  university,  where  the  program  had  been  implemented,  had 
an  additional  section  asking  questions  about  students’  awareness  and  understanding  of  the 
program  (see  the  questionnaire  at  the  end  of  this  part  of  the  example). 


d.  How  were  the  data  analyzed? 

Prior  to  data  entry,  the  questionnaires  had  to  be  coded.  Scores  reflecting  the  following  were 
required: 

1.  Alcohol  consumption 

2.  Attitudes  about  low-risk  drinking  practices 

3.  Knowledge  about  alcohol  use. 

Alcohol  consumption.  Alcohol  consumption  during  the  week  prior  to  testing  was  calculated 
from  the  retrospective  drinking  diary. 

Attitudes  about  low-risk  drinking  practices.  Attitudes  about  four  alcohol-related  and 
potentially  damaging  behaviours  were  examined,  with  attitudes  about  drinking  and  driving 
separated  into  personal  behaviour  and  others’  behaviour: 

1.  Drunkenness  (12  items  - max.  score  = 60) 

2.  Drinking  and  driving 

(a)  self  (4  items  - max.  score  = 20) 

(b)  others  (5  items  - max.  score  = 25) 

3.  Drinking  and  academics  (6  items  - max.  score  = 30) 

4.  Abstinence  (8  items  - max.  score  = 56). 
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Questions  1 through  27  were  used  to  examine  the  first  four  attitudes.  Each  question  was  scored 
as  follows: 

1 represented  the  most  positive  attitude  towards  alcohol  and  drinking  (i.e.,  in 
favour  of  alcohol  use);  5 represented  the  most  negative  attitude  towards  alcohol 
and  drinking  (i.e.,  against  alcohol  use). 

Question  28  examined  students’  attitudes  towards  abstinence,  and  was  scored  such  that  the 
highest  score  (i.e.,  7)  reflected  the  most  positive  attitude  towards  abstinence  for  each  scale,  and 
the  lowest  score  (i.e.,  1)  reflected  the  most  negative  attitude  towards  abstinence. 

Knowledge  about  alcohol  use.  Question  29  examined  the  students’  knowledge  about  alcohol 
use.  The  total  number  of  correct  answers  was  tabulated  (max.  score  = 13). 

Once  the  responses  were  coded,  the  data  were  entered  for  analysis  using  SPSS  (Statistical 
Package  for  Social  Sciences),  and  the  responses  were  matched  by  a respondent  code  consisting 
of  the  student’s  sex,  date  of  birth,  and  the  last  four  digits  of  their  parents’  telephone  number. 
Time  1 (before  the  program)  data  were  linked  with  data  from  time  2 (after  the  program)  for  each 
respondent.  The  primary  analysis  for  the  outcome  evaluation  was  a repeated- measures  analysis 
of  variance  (ANOVA)  that  compared  changes  in  knowledge  about  alcohol  drinking  practices  and 
changes  in  the  attitudes  of  students  who  were  exposed  to  the  program  (i.e.,  experimental 
university)  from  time  1 to  time  2 relative  to  changes  in  the  students  who  were  not  exposed  to 
the  program  (i.e.,  the  control  university)  from  time  1 to  time  2. 

Graphs  comparing  results  from  time  1 to  time  2 for  each  measure  were  prepared  using  Harvard 
Graphics  (see  Appendix  C). 


e.  What  did  they  find  out? 

Table  1 contains  a summary  of  the  seven  time  1 and  time  2 scores  for  experimental  and  control 
universities.  These  results  are  also  graphed  in  Figures  1-7. 

As  can  be  seen  in  Figure  1,  although  experimental  students  drank  more  than  those  at  the  control 
university  initially,  this  trend  was  reversed  at  time  2.  The  control  students  showed  a large 
increase  in  consumption  (a  finding  consistent  with  the  literature),  while  the  experimental  students 
showed  a slight  decrease. 

From  Figure  2 it  is  apparent  that  first  year  students’  attitudes  toward  drunkenness  generally 
became  slightly  more  positive  during  their  first  year.  There  was  no  significant  difference 
between  the  two  universities  on  this  variable. 
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In  Figure  3 it  can  be  seen  that  while  incoming  students  at  both  universities  had  similar  attitudes 
about  their  own  drinking  and  driving,  over  the  course  of  the  academic  term  their  attitudes 
diverged.  The  experimental  students  became  more  negative  and  the  experimental  students 
became  slightly  more  positive.  Attitudes  towards  drinking  and  driving  by  others  became  more 
negative  for  the  experimental  school  while  they  did  not  change  at  the  control  university  (see 
Figure  4). 

Regarding  drinking  and  academics,  students  at  the  control  university  showed  a more  positive 
attitude  toward  drinking  and  academics  over  time  while  the  shift  in  the  experimental  school  was 
toward  less  accepting  attitudes. 

As  shown  in  Figure  6,  respondents  at  the  experimental  university  became  more  positive 
regarding  abstinence  while  the  control  site  stayed  about  the  same.  The  acceptance  of  abstinence 
as  an  alternative  was  consistent  with  the  booklet  and  media  campaign. 

Finally,  Figure  7 shows  that  while  knowledge  increased  at  both  universities,  the  increase  at  the 
experimental  school  was  greater.  The  difference,  however,  was  not  statistically  significant. 
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Table  1.  Changes  in  Alcohol  Consumption,  Attitudes,  and  Knowledge  About  Alcohol 
Use  From  Time  1 to  Time  2 for  the  Experimental  and  Control  Universities 


Experimental  University 

Control  University 

Time! 

Time  2 

Time! 

Time  2 

Weekly  alcohol  consumption 
(F=7.3,p<.01) 

8.1 

7.7 

6.2 

9.2 

Attitudes  towards: 
drunkenness 
(F=2.7,  p<.10) 

40.0 

39.7 

41.5 

39.0 

drinking  and  driving  - self 
(F  = 8.5,p<.01) 

15.0 

15.8 

15.2 

14.8 

drinking  and  driving  - others 
(F=5.9,  p<.05) 

20.2 

21.4 

20.5 

20.8 

drinking  and  academics 
(F=7.6,  p<.01) 

20.6 

21.2 

21.0 

20.4 

abstinence 
(F=16.3,  p<.001) 

41.0 

45.0 

44.5 

42.5 

Knowledge  about  alcohol  and  drinking 
(F=2.8,  p<.10) 

6.6 

7.2 

6.2 

6.3 
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Figure  1 . Change  In  First  Year  Students'  Weekly  Alcohol 
Consumption  From  Time  1 to  Time  2 (Experimental 
University  Compared  with  Control  University) 
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Average  number  of  drinks  per  week 


Figure  2.  Change  in  First  Year  Students'  Attitudes  Towards 
Drunkenness  From  Time  1 to  Time  2 (Experimental 
University  Compared  with  Control  University) 


Average  attitude  score  (max=60) 
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Figure  3.  Change  in  First  Year  Students'  Attitudes  Towards 
Drinking  and  Driving  by  Themselves  From  Time  1 to  Time  2 
(Experimental  University  Compared  with  Control  University) 
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Average  attitude  score  (max=:20) 


Figure  4.  Change  in  First  Year  Students'  Attitudes  Towards 
Drinking  and  Driving  by  Others  from  Time  1 to  Time  2 
(Experimental  University  Compared  with  Control  University) 


Average  attitude  score  (max=25) 


University  Health  Promotion  Program 


Figure  5.  Change  in  First  Year  Students'  Attitudes 
Towards  Drinking  and  Academics  From  Time  1 to  Time  2 
(Experimental  University  Compared  with  Control  University) 
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Average  attitude  score  (max=30) 


Figure  6.  Change  in  First  Year  Students'  Attitudes  Towards 
Abstinence  From  Time  1 to  Time  2 (Experimental 
University  Compared  with  Control  University) 


Average  attitude  score  (max=56) 
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Figure  7.  Change  in  First  Year  Students'  Knowiedge  About  Aicohol 
and  Drinking  From  Time  1 to  Time  2 (Experimental 
University  Compared  with  Control  University) 


Average  knowledge  score  (max=10) 


/.  How  were  the  results  used? 

Despite  the  generally  positive  results  ultimately  demonstrated  by  the  evaluation  (in  terms  of 
knowledge,  attitudes  and  behaviour),  the  modifications  the  program  was  required  to  make  in  the 
design  and  samples  (see  Methodological,  measurement,  and  ethical  issues,  next  page)  meant  that 
the  results  needed  to  be  interpreted  cautiously.  In  addition,  the  demonstrated  short-term  positive 
effects  of  the  program  could  not  necessarily  be  assumed  to  be  long-lasting.  Although  the  results 
were  encouraging,  the  group  conducting  the  evaluation  concluded  that  further  research  is 
required  to  determine  how  long  the  impact  lasts  and  to  encourage  the  adoption  of  strategies  to 
promote  the  long-term  impact. 
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This  example  was  based  on  a large  scale  real  example  (Gliksman  et  al.,  1987), 
and  some  of  the  methodological  problems  encountered  in  the  study  provide 
some  lessons  in  the  typical  problems  encountered  in  real-life  evaluations. 


Methodological,  measurement,  and  ethical  issues: 

Methodological  and  Measurement.  Evaluations  are  conducted  in  the  real  world  and  often 
do  not  occur  quite  as  planned.  This  large  and  complex  project  encountered  a number  of 
problems  that  provide  an  interesting  lesson  in  the  difficulties  of  doing  evaluation.  The 
study  was  originally  designed  as  a randomized,  experimental,  field  study.  The  initial  plan 
was  to  have  access  to  the  registrars’  lists  of  incoming  first  year  students  at  both  the 
experimental  university  and  at  the  university  chosen  to  be  the  comparison  site.  From  these 
lists,  random  samples  of  1,500  students  were  to  be  selected  who  were  then  to  be  sent  a 
survey  by  mail  prior  to  their  arrival  at  the  university  (i.e.,  in  the  month  of  August).  One 
week  later,  all  subjects  were  to  be  sent  a reminder  letter.  One  week  after  this,  students  at 
the  experimental  site  were  to  be  sent  the  educational  material  by  the  university.  The 
material  was  sent  by  the  university  because  it  was  felt  that  students  would  be  more  likely  to 
read  the  material  if  it  was  perceived  as  being  part  of  the  initial  documentation  from  the 
university. 

Eight  months  later,  these  same  1,500  students  in  each  university  were  to  be  sent  a second 
survey,  which  would  serve  as  the  post-test  and  would  be  identical  to  the  first  except  for 
changes  that  would  accommodate  the  seven  months  they  would  have  been  on  campus.  The 
students  at  the  experimental  university  would  also  be  asked  about  their  awareness  of  the 
program  and  their  reactions  to  it. 

In  addition  to  these  questionnaires,  the  use  of  additional  data  sources  was  proposed.  To 
capture  the  effect  of  the  intervention  on  the  rest  of  the  student  population,  brief  interviews 
would  be  conducted  with  random  samples  of  second,  third,  and  fourth  year  students.  To 
determine  the  effects  of  alcohol  on  violent  acts  and  student  vandalism,  arrangements  were 
made  to  access  the  records  of  campus  police.  Finally,  to  determine  the  effects  of  the 
campaign  on  the  drinking  practices  of  students,  the  evaluators  were  to  be  permitted  to 
review  and  monitor  the  sales  data  of  the  university  bars  in  both  sites. 

Initial  arrangements  and  agreements  notwithstanding,  a number  of  events  occurred  that 
affected  the  results  and  necessitated  changes  in  both  the  implementation  of  the  program  and 
the  collection  and  analysis  of  the  data. 
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1.  Initially,  both  universities  agreed  to  provide  complete  lists  of  all  students  whom  they 
had  accepted  into  first  year.  Fifteen  hundred  students  were  to  be  randomly  selected  from 
these  complete  lists.  Only  the  experimental  site  provided  a complete  list  of  students.  The 
control  university  decided  to  select  the  1,500  students  on  its  own  and  send  those  names  and 
addresses.  Instead  of  randomly  selecting  these  students  from  the  entire  list,  it  was 
determined,  after  the  study  had  begun,  that  they  had  sent  the  first  1,500  names  on  their  list. 
Such  a sample  could  not  be  expected  to  reflect  the  population  of  first  year  students  at  that 
university,  since  those  accepted  first  are  likely  to  be  different  from  those  who  enrol  later. 

2.  It  was  anticipated  that  during  the  course  of  the  intervention,  a number  of  the  policy 
initiatives  which  had  been  accepted  in  principle  by  the  alcohol  advisory  committee  would  be 
implemented.  However,  because  of  the  reluctance  of  the  representatives  of  some  of  the 
departments  involved  in  the  sale  of  alcohol  who  were  on  the  alcohol  advisory  committee, 
these  were  never  adopted,  nor  even  formally  considered  for  implementation.  In  addition, 
one  of  the  policies  which  had  been  implemented,  differential  pricing,  was  the  cause  of  a 
great  deal  of  concern  for  the  managers  of  the  various  bars  on  campus  and  resulted  in  a 
number  of  confrontations.  Briefly,  the  policy  is  based  on  the  premise  that  students  could 
be  encouraged  to  drink  lower  alcohol-content  beverages  by  providing  a monetary  incentive 
- that  beer  be  priced  according  to  alcohol  content,  thus  making  2.5%  beer  less  expensive 
than  4.0%  beer  which  was  in  turn  less  expensive  than  5.0%  beer  which  was  then  less 
expensive  than  6.2%  beer.  Each  type  was  to  differ  in  price  by  $.25.  In  fact,  6.2%  beer 
was  already  more  expensive,  and  so  only  two  new  pricing  strategies  needed  to  be 
introduced.  Initially,  some  managers  resisted  the  quarter  difference  and  so  accommodations 
were  made  for  them.  Others  resisted  the  need  to  sell  the  very  light  beer  and  so  this 
requirement  was  dropped.  The  final  agreement  had  differential  pricing  for  6.2%,  5.0%, 
and  4.0%  beers,  with  one  bar  having  a quarter  differential,  a second  having  a fifteen  cent 
differential  and  a third  having  only  a ten  cent  differential.  After  two  months  of  operation, 
managers  refused  to  continue  with  the  differential  pricing  policy  because  they  claimed  that 
they  were  losing  too  much  money.  Although  the  alcohol  policy  committee  supported  the 
policy,  they  were  reluctant  to  confront  the  student  bar  managers  and  agreed  to  drop  the 
policy  completely,  reverting  to  the  initial,  uniform  pricing  policy  part  way  through  the 
year. 

3.  The  ads  which  appeared  in  the  student  newspaper  were  designed  to  promote  and 
reinforce  the  four  "appropriate"  behaviour  messages  initially  presented  in  the  booklet  sent 
out  by  the  university.  The  university  chose  not  to  have  its  name  associated  with  the 
messages  in  these  ads,  thus  effectively  disassociating  itself  from  the  theme  of  the  campaign 
and  the  booklet  initially  sent  out.  This  reduced  the  consistency  of  the  message  for  which 
the  program  had  strived. 
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4.  Because  the  lists  of  students  from  which  the  samples  of  1,500  were  drawn  were  not  the 
final  lists  of  students  who  attended,  but  only  those  who  had  been  accepted  and  who  had 
then  indicated  a willingness  to  attend,  a significant  proportion  of  students  in  each  school 
sample  (approximately  20%)  ultimately  did  not  attend  the  university  for  which  they  were 
drawn.  The  loss  of  subjects  for  the  study  was  further  exacerbated  because  the  surveys  were 
sent  to  their  parents’  homes  and  the  students  had  moved  or  were  working  out  of  town  for 
the  summer.  Thus,  some  students  did  not  receive  the  first  survey  or  the  education  package 
that  followed  two  weeks  later.  Also,  the  post-test  surveys  were  sent  to  the  initial  addresses 
of  students  applying  to  the  university,  and  many  surveys  were  returned  as  not  deliverable 
because  the  students  had  moved.  Finally,  some  students  chose  not  to  fill  out  surveys  at  all, 
some  would  not  fill  out  the  same  survey  twice,  and  others  were  unpredictable  about  when 
they  would  fill  out  a survey.  Therefore,  the  number  of  students  who  filled  out  both  the 
pre-test  and  the  post- test  was  lower  than  expected. 

5.  One  part  of  the  evaluation  included  monitoring  records  of  the  university  security 
departments.  It  had  been  indicated  that  these  departments  routinely  collect  information 
about  offenses  and  document  whether  these  charges  were  alcohol-related  or  had  another 
basis.  It  was  hoped  that  inspection  of  these  records  prior  to  the  intervention  and  subsequent 
to  the  intervention  would  indicate  whether  a decrease  in  alcohol-related  problems  involving 
these  departments  had  occurred.  Unfortunately,  the  records  provided  were  unusable.  They 
tended  to  be  incomplete  and  sporadic  in  the  frequency  of  their  completion,  and  invariably 
linked  the  offense  with  alcohol.  Thus,  one  anticipated  source  of  data  was  unavailable. 

6.  In  order  to  determine  whether  the  program  had  some  impact  on  the  drinking  practices 
of  students,  the  evaluation  was  to  include  examination  of  sales  records  of  the  individual 
campus  bars  at  both  sites  for  the  year  prior  to  the  beginning  of  the  campaign,  and  again  for 
the  period  during  which  the  intervention  was  in  place.  This  was  not  possible  because  the 
managers  of  the  experimental  bars  refused  to  provide  their  records  for  the  periods 
requested.  The  most  they  were  willing  to  provide  were  the  overall  sales  data  in  terms  of 
revenues  generated  and  not  in  terms  of  volume  and  types  of  beverages  sold,  which  was  the 
format  of  the  data  required  for  the  planned  analyses. 

The  analyses  which  were  initially  conducted  were  done  exactly  as  they  had  been  specified 
in  the  research  proposal.  They  did  not  accommodate  the  difficulties  and  problems 
encountered.  Not  surprisingly,  the  initial  set  of  results  showed  few  direct  impact  effects  of 
the  intervention.  However,  other  data  collected  from  the  general  student  population  and  the 
experimental  subjects  suggested  that  something  positive  was  happening.  It  was  found  that 
students  were  aware  of  the  program  and  that  they  recognized  the  themes  of  the  campaign. 
They  generally  agreed  with  the  policies  which  were  put  into  place  and  felt  that  there  had 
been  some  impact  on  their  behaviour. 
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Somehow,  the  inconsistency  between  the  positive  feedback  received  and  the  minimal 
program  benefits  observed  had  to  be  resolved.  Closer  inspection  of  the  data  revealed  that 
the  initial  messages  run  during  the  beginning  of  the  study  were  not  remembered,  while 
those  which  were  run  at  the  end  of  the  campaign,  closer  to  when  the  post- test  surveys  were 
sent  out,  were  remembered.  In  addition,  erosion  of  the  policies  took  place  in  the  two  to 
three  months  prior  to  the  post-test,  and  meant  that  whatever  impact  these  policies  may  have 
had  would  not  have  been  detected  because  they  were  not  in  place  at  the  time  of  the 
evaluation. 

To  overcome  some  of  these  problems  in  measuring  short-term  impact,  a second  intervention 
with  a modified  evaluation  was  planned  for  the  next  year.  The  second  intervention 
capitalized  on  the  lessons  learned  in  the  first  evaluation.  Two  major  changes  were  made  in 
the  intervention  itself.  First,  the  bar  managers  were  contacted  and  asked  to  provide  input 
into  the  policies  which  they  were  being  asked  to  support.  Only  those  policies  for  which 
unanimous  endorsement  could  be  achieved  were  implemented.  This  ensured  that  managers 
would  not  sabotage  policies  that  they  did  not  like,  but  also  meant  that  the  list  of  policies 
implemented  was  significantly  abbreviated  from  that  which  was  initially  implemented. 
Secondly,  the  media  campaign  was  conducted  over  a much  shorter  time  period.  The  four 
messages  were  run  during  a two-month  period  after  which  the  post- test  was  conducted. 
The  intent  then  was  to  determine  whether  the  program  had  any  short-term  impact.  The 
results  reported  in  Part  2 are  based  on  this  modified  evaluation. 

Ethical.  Maintaining  the  confidentiality  of  the  information  collected  as  part  of  the 
evaluation  was  of  paramount  importance.  In  order  to  follow  up  students  for  the  second 
testing,  it  was  necessary  to  retain  their  names  and  addresses.  In  order  to  ensure  the 
confidentiality  of  questionnaires,  pre-test  and  post-test  questionnaires  were  linked  using 
information  that  did  not  reveal  the  identity  of  respondents  in  the  datafile,  including: 
gender,  date  of  birth,  and  the  last  four  digits  of  the  respondent’s  parents’  telephone 
number.  Thus,  although  questionnaires  were  mailed  using  names,  they  came  back 
anonymously. 
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UNIVERSITY  SURVEY 


What  is  your  birthdate? 

Day  Month  Year 

What  are  the  last  4 digits  of  your  home 

(parents’)  phone  number? 

Please  indicate  whether  you  are  male  or  female: 

Male  Female  

The  following  set  of  questions  deal  with  some  of  your  attitudes  and  beliefs  about  alcohol  use  and 
alcohol  problems.  Please  circle  the  number  to  the  right  of  the  question  that  most  accurately 
reflects  your  attitude. 


Disagree  Strongly  Disagree  Neutral/Not  sure  Agree  Strongly  Agree 


1 2 3 4 5 

1.  ITl  probably  not  refrain  from  driving  if  I’ve  had  one  or  two  drinks.  1 2 

2.  I don’t  enjoy  parties  when  everyone  gets  really  drunk.  1 2 

3.  It’s  my  responsibility  not  to  let  a friend  who’s  impaired  drive.  1 2 

4.  If  you’re  feeling  depressed  about  too  much  school  work,  the  best  1 2 

thing  to  do  is  relax  and  have  a few  drinks. 

5.  I think  I am  a more  alert  driver  if  I’ve  had  a small  amount  to  drink.  1 2 

6.  It’s  not  a good  idea  to  have  a drink  or  two  at  dinner  if  you  have  an  12 

evening  class. 

7.  The  best  parties  are  the  ones  I can’t  remember  very  well.  1 2 

8.  A hangover  is  a reasonable  price  to  pay  for  having  a good  time  at  a party.  1 2 

9.  People  can  judge  for  themselves  whether  they’re  too  drunk  to  drive.  1 2 

10.  I probably  wouldn’t  date  someone  who  gets  drunk  a lot  at  parties.  1 2 

1 1 . Friends  would  think  I couldn’t  handle  my  liquor  if  I left  my  car  and  1 2 

took  a taxi  home. 


3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 


4 5 
4 5 
4 5 
4 5 

4 5 
4 5 

4 5 
4 5 
4 5 
4 5 
4 5 
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12.  No  doubt  I will  hang  around  with  a crowd  that  gets  fairly  drunk  at 
pubs  and  parties. 

13.  Getting  drunk  is  a problem  only  if  you  hurt  yourself  or  somebody  else. 

14.  I expect  I’ll  have  a few  drinks  if  I feel  uptight  about  school. 

15.  I will  have  a better  time  at  a party  if  I get  drunk. 

16.  I don’t  enjoy  hanging  around  with  people  who  get  drunk. 

17.  The  host  at  a party  should  make  sure  that  anyone  who  is  drunk 
doesn’t  drive  home. 

18.  I will  probably  get  really  drunk  at  a few  parties  this  year. 

19.  There’s  not  much  point  to  a party  if  you  don’t  get  fairly  hammered. 

20.  I would  make  my  friends  stay  overnight  or  take  a cab  or  bus  home  if 
they  got  drunk  at  my  place. 

21.  There’s  nothing  wrong  with  getting  drunk  at  a party. 

22.  Jogging  or  some  other  form  of  exercise  is  more  relaxing  than 
drinking  if  you’re  uptight  at  school. 

23.  There’s  nothing  wrong  with  having  a few  drinks  before  studying. 

24.  There’s  nothing  wrong  with  getting  drunk  occasionally. 

25.  Knowing  myself,  I’d  drive  even  if  I’d  had  a lot  to  drink. 

26.  A small  amount  of  alcohol  before  an  exam  can  make  you  less 
nervous  so  you’ll  do  better. 

27.  I think  drunks  at  parties  are  pretty  entertaining. 


1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 
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28.  Please  place  an  ’X’  in  the  space  on  the  scale  that  best  reflects  your  impressions  about 
abstinence  or  not  drinking. 


(a)  pleasant 

unpleasant 

(b)  unsociable 

sociable 

(c)  immoral 

moral 

(d)  unhealthy 

healthv 

(e)  fair 

unfair 

(f)  foolish 

wise 

(g)  good 

bad 

(h)  appropriate 

inappropriate 

29.  In  this  section  we  would  like  to  ask  you  some  more  information  about  alcohol.  Please 
print  the  letter  of  the  selected  alternative  in  the  space  to  the  left  of  each  question. 

(a)  True  (b)  False  (c)  Don’t  Know 

a.  One  can  be  impaired  without  being  drunk. 

b.  Mixed  drinks  aren’t  absorbed  by  the  body  as  quickly  as  undiluted  drinks. 

c.  More  than  50%  of  the  adult  population  drink  less  than  two  standard  drinks  per  day. 

d.  One  to  two  drinks  wouldn’t  impair  co-ordination  in  a young  healthy  person. 

e.  Alcohol  can  cause  irreversible  brain  damage. 

f.  An  average  of  four  drinks  per  day  or  more,  over  a long  time,  will  not  increase  your 

risk  of  health  problems. 

g.  There  is  no  evidence  that  alcohol  consumption  increases  risk  of  cancer. 

h.  One  beer  won’t  make  you  as  drunk  as  a shot  of  hard  stuff  like  whisky. 

i.  If  your  stomach  is  full,  you  can  drink  as  much  as  you  want  without  getting  drunk. 

j.  Ten  percent  of  college  students  can  be  classified  as  heavy  drinkers. 

k.  A person  cannot  be  charged  with  impaired  driving  if  his  or  her  blood  alcohol  level 

is  under  .05. 

1.  The  limit  on  any  one  occasion  should  be  no  more  than  four  drinks. 

m.  Alcohol  has  the  same  effect  on  men  and  women  if  their  weights  are  similar. 
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Drinking  Diary 


Please  estimate  the  number  and  type  of  alcoholic  drinks  that  you  had  for  each  of  the  days 
during  the  past  week. 


DAY 

Amount  & Type  of  Beverage 

How  many  bottles 
of  regular  beer? 
(5%  ale) 

How  many  bottles 
of  light  beer? 
(<5%  ale) 

How  many 

1 V2  oz.  glasses 
of  spirits? 

How  many  5 oz. 
glasses  of  wine? 

SUNDAY 

MONDAY 

TUESDAY 

WEDNESDAY 

THURSDAY 

FRIDAY 

SATURDAY 
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C.  Parent  Education  Workshop: 

A Health  Promotion  Presentation  about 
Alcohol  and  Drugs  for  Parents  of  Adolescent  Children 


Program:  The  program  was  sponsored  by  the  Drug  and  Alcohol  Committee  of  a local  school 
board  and  was  targeted  at  parents  of  adolescent  students.  The  committee,  hoping  to  raise 
awareness  among  parents,  presented  an  evening  panel  discussion  of  drug  and  alcohol  issues 
confronting  adolescents. 

This  example  includes  quality  enhancement  questions  for  three  levels  of  evaluation: 


Part  1.  Consumer  Characteristics 

Part  2.  Consumer  Satisfaction 

Part  3.  Consumer  Characteristics  by  Consumer  Satisfaction. 
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Level  of  difficulty:  easy 

Level  of  resources  needed:  minimal 


Part  1.  Consumer  Characteristics 

What  proportion  those  who  attended  were  parents  of  adolescent 
children? 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 

Committee  members  noticed  that  the  majority  of  people  who  attended  last  year  were  mothers 
even  though  invitations  had  been  extended  to  both  parents.  Since  the  committee  feels  that  it  is 
helpful  to  have  both  parents  aware  of  the  issues,  they  were  interested  in  why  only  mothers  were 
attending. 

A concern  of  the  Drug  and  Alcohol  Committee  was  that  the  information  they  were  presenting 
was  designed  for  the  parents  of  adolescent  children  who  may  already  be  experimenting  with 
drugs.  From  the  questions  asked  at  last  year’s  presentation,  it  seemed  that  many  of  the  audience 
were  parents  of  children  still  in  public  school.  If,  indeed,  the  majority  of  the  audience  were 
parents  of  pre-adolescents,  an  effort  should  be  made  to  include  information  for  parents  of 
younger  children  who  may  not  yet  be  experimenting. 


h.  What  resources  were  needed  to  collect  and  interpret  the  data? 

It  was  necessary  to  design  a short  questionnaire  to  be  completed  by  members  of  the  audience 
(see  questionnaire  at  the  end  of  Part  1).  This  took  approximately  two  days  including  meeting 
to  decide  on  the  questions,  drafting  the  questions,  typing  and  duplicating  the  questionnaire.  It 
also  took  a committee  member  a couple  of  hours  to  tally  the  information  from  the  questionnaires 
and  prepare  a summary. 
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c.  How  were  the  data  collected? 

Following  the  presentation,  the  members  of  the  audience  were  asked  to  remain  in  their  seats  in 
order  to  complete  a short  evaluation  questionnaire.  Volunteers  handed  a questionnaire  to  each 
person  and  stood  by  the  door  to  collect  completed  surveys. 


d.  How  were  the  data  analyzed? 

Questions  2,  4,  5 and  6 of  the  questionnaire  were  used  to  address  the  quality  enhancement 
questions  specified  for  this  part  of  the  example.  A committee  member  totalled  responses  from 
question  2 to  identify  the  number  of  males  and  females  attending.  Questions  4 and  6 were 
totalled  by  hand  in  the  same  way.  For  question  5,  the  number  of  children  reported  by  each 
participant  was  calculated  and  divided  by  the  number  of  participants  to  arrive  at  an  average  score 
for  number  of  children. 


e.  What  did  they  find  out? 

All  60  people  who  attended  the  presentation  completed  the  feedback  questionnaire.  Of  these, 
48  (80%)  were  women.  Of  the  48  women  who  attended,  37  (77%)  indicated  that  they  were 
separated,  divorced  or  widowed.  Of  the  12  men  who  attended,  only  two  (17%)  indicated  that 
they  were  separated,  divorced  or  widowed.  Since  some  people  who  attended  were  married 
couples,  the  analyses  addressing  the  second  question  (proportion  of  those  attending  who  were 
parents  of  adolescents)  were  based  on  households,  not  individual  respondents.  There  were  six 
married  couples  who  attended;  therefore,  responses  for  54  households  were  included  in  the 
analyses.  The  average  number  of  children  per  household  among  those  attending  was  2.7. 
Twenty-four  households  (44%)  had  children  who  were  13  or  older,  23  households  (43%)  had 
children  12  or  under,  and  seven  households  (13%)  had  at  least  one  teenaged  child  and  at  least 
one  younger  child. 


/.  How  were  the  results  used? 

The  information  collected  was  useful  to  the  committee  for  planning  next  year’s  evening  panel 
discussion.  Since  many  of  the  parents  were  single  mothers  it  was  decided  to  address  some  issues 
facing  single  parents  who  were  dealing  with  their  children’s  drug  use.  Also,  because  parents 
of  pre-adolescents  were  attending  the  discussion,  it  was  decided  to  include  issues  facing  them 
and  their  children  as  well. 
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DRUG  AND  ALCOHOL  AWARENESS  AND  YOUR  TEENS 

Panel  Discussion 


We  are  interested  in  your  reactions  to  the  panel  discussion  on  "Drug  and  Alcohol 
Awareness  and  Your  Teens".  Please  complete  this  questionnaire  to  help  us  plan  future 
Alcohol  and  Drug  Awareness  panel  discussions.  Your  responses  will  be  totally  anonymous. 
Thank  you  for  your  cooperation. 


1 . What  is  your  age?  | | | 

2.  Are  you?  □ Male 

□ Female 

3.  What  is  your  current  marital  status? 

□ Married  or  living  common-law 

□ Separated,  divorced,  or  widowed 

□ Single,  never  married 


4.  How  many  children  do  you  have?  

5.  What  are  the  ages  of  your  children?  (Check  one  only.) 

□ 13  or  older 

□ 12  or  younger 

□ Both  13  or  older  and  12  or  younger 

6.  How  did  you  hear  about  this  event?  (Check  all  that  apply.) 


□ 

At  work 

□ 

Media  advertisement 

□ 

At  church 

□ 

Word  of  mouth 

□ 

At  school 

□ 

Other  (please  describe) 

□ 

Children’s  school 
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Why  did  you  attend?  (Check  all  that  apply.) 

□ General  interest 

□ Worried  about  my  kids  now 

□ Worried  about  my  kids  when  they  are  teens 


Please  rate  this  panel  discussion  on  the  following  scales  by  placing  an  X in  the  space 
closest  to  the  word  that  best  describes  your  feelings. 


8. 


The  Panel  Discussion  was: 

informative  | | | | 

useful  I I I I 

organized  | | | | 

interesting  | | | | 

clear  | | [ | 

helpful  I I i I 

understandable  I I I I 


uninformative 

useless 

disorganized 

boring 

confusing 

not  helpful 

not  understandable 


9.  Considering  how  you  felt  about  this  panel  discussion,  would  you  attend  another  Drug  and 
Alcohol  Awareness  panel  discussion  if  one  were  held  in  the  future? 

□ Yes 

□ No 


10.  Further  comments  or  suggestions  for  improvement 
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Level  of  difficulty:  easy 

Level  of  resources  needed:  moderate 


Part  2.  Consumer  Satisfaction 


How  satis^ed  were  peopMMhd  attend  presentation? 


a.  Who  was  asking  the  question  and  why  did  they  want  this  information? 

When  a similar  program  was  held  last  year,  a few  members  of  the  audience  indicated  to 
committee  members  that  they  enjoyed  the  evening.  However,  most  people  did  not  bother  to 
provide  either  positive  or  negative  comments.  This  year,  the  committee  was  interested  in  finding 
out  in  a more  systematic  way  whether  the  program  they  presented  was  well  received  by  those 
who  attended.  They  were  also  interested  in  ways  the  audience  felt  the  program  could  be 
improved  for  next  year. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

One  committee  member  spent  one  day  coding  the  information  required,  entering  some  of  the 
information  into  the  computer,  tallying  some  information  from  the  questionnaires,  and  preparing 
a summary. 


c.  How  were  the  data  collected? 

All  information  required  had  previously  been  collected  (see  questionnaire.  Part  1). 


d.  How  were  the  data  analyzed? 

The  data  from  questions  9 and  10  of  the  questionnaire  used  in  Part  1 were  analyzed  using  Lotus 
1-2-3  (see  Appendix  B).  Participants  had  been  asked  to  respond  to  question  9 by  putting  an  "X" 
in  the  appropriate  space.  Before  entering  the  data  into  the  computer  it  was  necessary  to  convert 
the  X’s  in  the  spaces  to  numbers.  Therefore,  the  first  space  on  the  left  was  given  a value  of  5; 
the  second  space,  a value  of  4;  the  middle  space,  a value  of  3;  the  fourth  space,  a value  of  2; 
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and  fifth  space,  a value  of  1 and  each  respondent’s  ratings  numbered  as  shown  in  the  example 
in  Figure  1.  For  this  respondent,  the  numbers  4,  5,  3,  1,4,  5,  and  2 would  be  entered  into  the 
computer.  Once  the  data  were  entered  for  all  respondents,  means  (i.e.,  average  scores)  were 
calculated  for  ratings  of  each  aspect  of  the  presentation  across  respondents. 


Figure  1.  Example  of  Coding  for  One  Person’s  Responses  to  Question  9 


informative 

useful 

organized 

interesting 

clear 

helpful 

understandable 


The  Panel  Discussion  was: 


I 

I 


I 

I 


uninformative 

useless 

disorganized 

boring 

confusing 

not  helpful 

not  understandable 


Responses  to  question  10  were  also  entered  into  Lotus  1-2-3  and  the  frequency  of  Yes  responses 
computed.  Responses  to  question  11  were  summarized  by  hand. 

A bar  graph  illustrating  the  results  was  prepared  using  Lotus  1-2-3  (see  Appendix  B). 


e.  What  did  they  find  out? 

All  60  people  who  attended  the  presentation  completed  the  feedback  questionnaire.  As  shown 
in  Table  1,  respondents  rated  the  panel  discussion  very  highly  on  most  dimensions.  The  most 
highly  rated  dimensions  were  helpfulness,  informativeness,  and  usefulness.  The  lowest  rating 
was  given  to  the  extent  to  which  the  presentation  had  been  well-organized.  Figure  2 shows  a 
bar  graph  of  the  same  results,  prepared  using  Lotus  1-2-3. 

The  presentation  had  been  late  starting  and  there  was  some  confusion  in  running  the  panel 
demonstration.  When  asked  for  suggestions  for  making  the  evening  better  next  year,  12  people 
suggested  that  the  program  begin  on  time,  seven  people  said  that  some  panel  members  should 
speak  more  slowly,  and  seven  people  requested  that  there  be  some  information  or  pamphlets  to 
take  home.  Fifty  (83%)  respondents  reported  that  they  would  be  willing  to  attend  another  panel 
discussion. 
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1.  Average  Ratings  of  Various  Aspects  of  the  Panel  Discussion 


Score 

Average  satisfaction  ratings: 

informative 

4.7 

useful 

4.6 

organized 

2.9 

interesting 

4.2 

clear 

4.0 

helpful 

4.8 

understandable 

4.1 

Figure  2.  Average  Ratings  of  Various  Aspects  of  the  Panel  Discussion 

1 2 3 4 5 


Informative 


Understandable 
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/.  How  were  the  results  used? 

The  information  collected  was  useful  to  the  committee  for  planning  next  year’s  presentation. 
They  decided  to  retain  the  evening  panel  discussion,  but  to  ensure  that  it  would  start  on  time. 
The  presenters  would  be  briefed  ahead  of  time  regarding  the  speed  with  which  material  should 
be  presented  and  the  format  of  the  discussion.  In  addition,  presenters  would  be  asked  to  prepare 
handouts  that  could  be  distributed  to  those  who  attended.  The  committee  members  also  planned 
to  ensure  that  pamphlets  on  alcohol  and  drug  use  by  young  people  would  be  available. 
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Level  of  difficulty:  easy 

Level  of  resources  needed:  moderate 


Part  3.  Consumer  Characteristics  by  Consumer  Satisfaction 


Was  there  a difference  between  mothers  and  fathers  on  how  satisfied  they 
were  with  the  presentation? 

Did  parents  of  adolescent  children  rate  the  presentation  more  positively 
than  parents  of  younger  children? 


a.  Who  was  asking  the  question  and  why  did  they  want  this  information? 

Because  more  mothers  than  fathers  attended  the  presentation,  the  committee  wanted  to  be  sure 
that  there  was  nothing  in  the  content  of  the  program  that  was  causing  men  to  stay  away.  Also, 
because  the  committee  was  aware  that  parents  of  pre-adolescent  children  were  attending  the 
presentation,  they  were  interested  in  their  ratings  of  the  evening  compared  to  the  ratings  by 
parents  of  adolescents,  for  whom  the  presentation  had  been  designed  (see  Parts  1 and  2 of  this 
example). 


b.  What  resources  were  needed  to  collect  and  analyze  the  information? 

A committee  member  had  already  entered  much  of  the  relevant  information  on  the  computer 
using  Lotus  1-2-3.  About  three  hours  were  needed  to  do  the  appropriate  analyses  and  prepare 
summaries  of  the  results. 


c.  How  were  the  data  collected? 

All  information  required  had  previously  been  collected  (see  questionnaire.  Part  1). 
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The  data  for  questions  9 and  10  had  been  entered  into  the  computer  previously.  [Typically,  all 
information  would  have  been  entered  into  the  computer  in  the  first  place.  This  example 
has  been  designed,  however,  to  demonstrate  how  an  evaluation  can  be  broken  down  into 
simple  stages  and  to  demonstrate  adding  new  data  to  an  existing  file.]  It  was  now  necessary 
to  add  the  information  from  questions  2 and  6 so  that  relationships  between  satisfaction  and 
gender  and  satisfaction  and  age  of  children  could  be  examined.  As  in  question  9,  the  check 
marks  in  questions  2 and  6 had  to  be  converted  to  numbers  before  being  entered  into  the 
computer.  Therefore,  in  question  2,  ‘male’  was  given  a value  of  1 and  ‘female’  a value  of  2. 

For  ease  of  analysis,  it  was  decided  to  combine  the  people  who  only  had  children  over  13  with 
those  who  had  children  both  13  or  older  and  12  or  younger  (i.e.,  combining  all  those  who  had 
at  least  one  teen).  Therefore,  respondents  were  coded  as  1 - ‘parent  of  at  least  one  child  who 
is  13  or  older’  and  2 - ‘parent  of  a child  or  children  who  are  all  12  or  younger’. 

Means  were  compared  for  the  seven  five-point  scales  in  question  9 for  male  compared  to  female 
participants  and  for  parents  of  the  two  age  groups  of  children.  In  this  part  of  the  example,  the 
fact  that  some  respondents  were  married  to  each  other  was  ignored. 

Line  graphs  comparing  the  results  for  each  of  the  two  comparisons  were  prepared  using  Lotus 
1-2-3  (see  Appendix  B). 


e.  What  did  they  find  out? 

As  shown  in  Table  1,  both  males  and  females  rated  the  panel  discussion  positively,  with  slightly 
higher  ratings  provided  by  females  on  all  aspects  of  the  presentation.  The  same  information  is 
presented  in  Figure  1 in  line  graph  format,  using  Lotus  1-2-3. 

Thirty-five  people  who  attended  the  presentation  had  at  least  one  child  13  years  of  age  or  over, 
while  25  people  reported  that  all  their  children  were  12  and  under.  There  were  no  major 
differences  in  ratings  of  parents  of  teens  versus  those  whose  children  were  under  13,  although 
there  was  a trend  for  those  with  teenage  children  to  rate  the  program  slightly  more  highly  (see 
Table  2 and  Figure  2). 
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Table  1.  Average  Ratings  of  Various  Aspects  of  the  Panel  Discussion  for  Females 
Versus  Males  Who  Attended  the  Presentation 


Females 

n=48 

Males 

n=12 

informative 

4.7 

4.6 

useful 

4.6 

4.3 

organized 

3.0 

2.7 

interesting 

4.3 

3.9 

clear 

4.1 

3.5 

helpful 

4.8 

4.7 

understandable 

4.2 

3.7 

Figure  1.  Average  Ratings  of  Various  Aspects  of  the  Panel  Discussion 
for  Females  Versus  Males  Who  Attended  the  Presentation 


Informative 

Organized  Clear 

Understandable 

Useful 

Interesting 

Helpful 

Rating  Category 

^ Male 


Female 
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Table  2.  Average  Ratings  of  Various  Aspects  of  the  Panel  Discussion  by  Parents  of 
Teens  Versus  Parents  Whose  Children  Were  All  Under  Age  13 


All  children  aged 

12  or  younger 
n=25 

At  least  one 
child  13  or  older 
n=35 

informative 

4.6 

4.8 

useful 

4.5 

4.7 

organized 

2.8 

3.1 

interesting 

4.1 

4.3 

clear 

3.9 

4.2 

helpful 

4.8 

4.8 

understandable 

4.1 

4.1 

Figure  2.  Average  Ratings  of  Various  Aspects  of  the  Panel  Discussion  by  Parents  of 


Informative  Organized  Clear  Understandable 

Useful  Interesting  Helpful 

Rating  Category 

a At  least  one  adolescent  child  No  adolescent  children 
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/.  How  were  the  results  used? 


It  appeared  that  fathers  who  attended  the  presentation  were  slightly  less  satisfied  than  mothers. 
Since  the  differences  were  not  large  and  the  number  of  fathers  who  attended  was  small,  no  firm 
conclusions  could  be  drawn  from  the  differences  observed.  The  committee  assumed  that  the 
high  proportion  of  mothers  attending  was  probably  a product  of  the  culture,  but  that  they  should 
continue  to  monitor  satisfaction  ratings  of  fathers  compared  to  mothers. 

Parents  appeared  to  be  equally  satisfied  with  the  presentation  regardless  of  the  ages  of  their 
children,  although  ratings  by  parents  whose  children  were  all  under  13  were  slightly  lower.  This 
seemed  to  indicate  that  the  material  was  appreciated  by  parents  regardless  of  whether  their 
children  had  reached  their  teen  years  yet. 
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D.  Workplace  Health  Promotion  Program 


Program:  A large  car  manufacturing  plant  developed  and  evaluated  a health  promotion 
program  that  included  three  components:  a smoking  cessation  program,  a fitness  program,  and 
a blood  pressure  clinic.  The  smoking  cessation  program  provided  education  on  aversive 
strategies  and  self-control  strategies  used  to  overcome  urges  to  smoke,  and  was  delivered  by  an 
external  program  leader  paid  by  the  company  on  a fee-per- service  basis.  Because  of  its  cost  and 
format,  it  was  decided  to  offer  the  smoking  cessation  program  to  only  half  of  the  smokers  in  the 
company.  Eligibility  for  the  program  would  be  determined  by  random  selection  using  employee 
numbers  for  those  who  reported  being  light  or  heavy  smokers  at  pre-test.  This  would  allow  an 
unbiased  determination  of  the  effectiveness  of  the  smoking  cessation  program  and  provide  a 
suitable  design  for  evaluating  cost  effectiveness.  The  program  consisted  of  eight  sessions  (50 
minutes  each)  that  occurred  at  the  beginning/end  of  shifts  (using  30  minutes  from  each  shift). 
Thus,  those  attending  were  allowed  to  miss  30  minutes  of  work  (either  at  the  beginning  or  end 
of  their  shifts)  for  which  they  were  paid.  Eligible  employees  could  sign  up  for  only  one  set  of 
eight  sessions. 

The  fitness  program  consisted  of  a 50-minute  workout  offered  everyday  at  noon.  The  blood 
pressure  clinic  involved  a nurse  being  available  to  check  employees’  blood  pressure.  The  nurse 
provided  education  regarding  the  importance  of  addressing  high  blood  pressure  and  encouraged 
employees  to  see  their  family  physician  if  their  blood  pressure  exceeded  140/90  on  at  least  two 
occasions. 

This  example  includes  quality  enhancement  questions  for  six  levels  of  evaluation: 


Part  1. 

Logic  Model 

Part  2. 

Client  Characteristics 

Part  3. 

Program  Delivery 

Part  4. 

Outcome  Evaluation 

Part  5. 

Program  Delivery  by  Outcome  Evaluation 

Part  6. 

Economic  Evaluation. 
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Part  1.  Logic  Model 


Logic  Model  for  Company  Health  Promotion  and  Prevention  Program 


MAIN 

COMPONENTS 

SMOKING 

CESSATION 

PROGRAM 

FITNESS 

PROGRAM 

BLOOD 

PRESSURE 

CLINIC 

4 

1 

1 

IMPLEMENTATION 

OBJECTIVES 

To  teach  self- 
control  strategies  to 
people  trying  to 
stop  smoking 

To  provide  Htness 
opportunities  to 
employees 

To  provide 
education  to 
employees  about 
reducing  blood 
pressure 

1 

1 

OUTPUTS 

# of  course 
participants 

# of  participants 
in  Htness  programs 

# of  employees 
who  had  blood 
pressure  measured 

1 

1 

SHORT-TERM 

OUTCOME 

OBJECTIVES 

To  reduce 
smoking  among 
employees 

To  increase 
cardiovascular 
health  of  employees 

To  reduce  blood 
pressure  of 
employees 

1 

OUTCOME 

INDICATORS 

# of  employees 
who  quit  or  reduce 
smoking 

# of  employees 
who  exhibit 
improved  Htness 
levels 

it  of  employees 
whose  blood 
pressure  was 
reduced  from 
problem  range  to 
non-problem  range 

\ 

i 

LONG-TERM 

OUTCOME  OBJECTIVES 

To  increase  the 
overall  health  of 
employees 

1 

LONG-TERM 

OUTCOME  INDICATORS 

% of  employees  whose  overall 
health  improved 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  heavy 


Part  2.  Consumer  Characteristics 


What  proportion  of  employees  were  smokers  at  the  beginning  of  the 
program? 

What  were  tMfithMs  when  the  program  began? 

What  proportion  (f  et^  classified  as  hypertensive  (i,e,j  blood 

pressure  reading  of  140/90y  at  the  beginning  of  the  program? 


a.  Who  was  asking  the  questions  and  why  did  they  want  the  information? 

The  company  has  developed  a workplace  health  promotion  program,  including  three 
components:  a smoking  cessation  program,  a fitness  program,  and  a blood  pressure  clinic.  Part 
of  implementing  the  program  included  assessing  the  initial  smoking  status,  fitness  level,  and 
blood  pressure  level  of  the  employees.  This  information  would  serve  as  a baseline  for  assessing 
the  impact  of  the  program. 

Employees  were  given  time  off  to  participate  in  the  assessment.  Education  regarding  blood 
pressure  was  provided  at  the  time  of  the  testing.  Participation  was  not  required,  but  employees 
were  encouraged  to  attend,  and  most  did. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

The  five  company  nurses  spent  approximately  two  months  each  assessing  employees  and 
documenting  the  information  on  a summary  form.  The  forms  were  sent  to  a key-punching  firm 
for  data  entry.  The  data  were  returned  on  disks  to  the  work-force  health  co-ordinator  who  spent 
approximately  15  hours  cleaning  the  data  file  for  data  entry  errors  and  analyzing  the  information. 
In  order  to  complete  the  fitness  test,  a treadmill  was  required.  In  order  to  complete  the  analyses 
for  this  evaluation,  a computer  and  software  capable  of  computing  simple  statistics  were 
required. 
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c.  How  were  the  data  collected? 


As  a part  of  the  assessment  process,  the  company  nurses  completed  an  assessment  summary  for 
every  employee.  Each  employee  was  assessed  anonymously,  but  a code  was  developed  to  serve 
as  an  identifier  for  future  reference.  This  code  consisted  of  the  last  four  digits  of  each 
employee’s  phone  number,  his  or  her  day  of  birth  (2  digits),  and  month  of  birth  (2  digits  - 01 
to  12).  Information  about  employees  was  coded  as  follows: 


(a)  smoking  status: 

1.  non-smoker  (never  smoked  or  has  not  smoked  for  at  least  12  months) 

2.  recent  former  smoker  (quit  smoking  within  the  last  12  months) 

3.  light  smoker  (less  than  10  cigarettes  per  day) 

4.  heavy  smoker  (10  or  more  cigarettes  per  day) 


(b)  fitness  level: 

1.  very  fit 

2.  adequate  fitness 

3.  less  than  adequate  fitness 


(c)  blood  pressure: 

1.  non-hypertensive  (BP  = 140/90  or  less) 

2.  hypertensive  (BP  with  higher  number  greater  than  140  and/or  lower 
number  greater  than  90,  tested  on  at  least  2 occasions). 


Fitness  level  was  determined  by  measuring  body  weight,  body  fat,  resting  heart  rate,  and  heart 
rate  at  two  intervals  during  treadmill  exercise. 


d.  How  were  the  data  analyzed? 

Proportions  of  employees  in  each  category  of  smoking  status,  fitness  level,  and  blood  pressure 
were  calculated  using  a computer.  Pie  graphs  illustrating  the  results  were  prepared  using 
Harvard  Graphics  (see  Appendix  C). 
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Figure  1.  Smoking  Status  of  Employees  at  Initial  Assessment 


Non-smoker 


(2,986)  64% 


(293)  6%  Light  smoker 

(492)  10% 


Figure  2.  Fitness  Level  of  Employees  at  Initial  Assessment 


Not  adequately  fit 
(2,761)  59% 


Very 

(290) 


fit 
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Adequately  fit 
(1,649)  35% 
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Figure  3.  Blood  Pressure  of  Employees  at  Initial  Assessment 


e.  What  did  they  find  out? 

Of  5,000  employees,  4,700  (94%)  participated  in  the  program.  While  the  majority  of  employees 
were  non-smokers,  20%  were  heavy  smokers  and  an  additional  10%  were  light  smokers  (see  pie 
graph.  Figure  1,  prepared  using  Harvard  Graphics).  As  shown  in  Figure  2,  the  majority  of 
employees  (59%)  were  less  than  adequately  fit,  35%  were  adequately  fit,  and  6%  were  very  fit. 
Finally,  as  Figure  3 indicates,  20%  of  the  employees  were  categorized  as  having  hypertensive 
blood  pressure  levels. 


/.  How  were  the  results  used? 

The  results  were  kept  in  the  computer  to  serve  as  a baseline  for  comparison  with  information 
to  be  gathered  one  year  after  the  program  had  been  in  place. 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  moderate 

Part  3.  Program  Delivery 

What  proportidh  of  employees  program? 

What  proportion  of  eligible  employees  participated  in  the  smoking 
cessation  program  ? 


a.  Who  was  asking  the  questions  and  why  did  they  want  the  information? 

As  part  of  implementing  the  programs,  the  company  evaluated  attendance  at  the  programs  at  the 
end  of  one  year.  In  particular,  they  wished  to  determine  the  proportion  of  all  employees  who 
attended  the  fitness  program  as  well  as  the  proportion  of  eligible  light  or  heavy  smoking 
employees  who  attended  the  smoking  cessation  program. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

The  smoking  cessation  program  leader  maintained  ongoing  records  of  attendance  in  the  program. 
The  fitness  program  included  a system  of  tracking  attendance.  The  work-force  health  co- 
ordinator spent  approximately  four  hours  analyzing  the  information.  In  order  to  complete  the 
analyses  for  this  evaluation,  a computer  and  software  capable  of  computing  simple  statistics  were 
required. 


c.  How  were  the  data  collected? 

The  smoking  cessation  program  leader  maintained  a count  of  all  employees  who  used  the 
program.  This  information  was  provided  to  the  work-force  health  co-ordinator.  Participants  in 
the  fitness  program  were  asked  to  swipe  their  employee  card  each  time  they  used  the  fitness 
facility.  Information  about  usage  of  the  facility  was  tracked  by  the  company  computer  and  the 
computer  output  was  provided  to  the  work-force  health  co-ordinator. 
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The  proportion  of  eligible  employees  of  the  company  who  had  used  the  smoking  cessation 
program  within  the  last  12  months  was  calculated  by  hand.  In  addition,  the  total  number  of 
participants  in  each  session  was  summed  across  all  sessions  to  determine  the  total  number  of 
employee  sessions  provided.  Using  the  computer  file  on  attendance  at  the  fitness  program, 
employees  were  classified  as:  non-attenders  or  attended  less  than  twice  a month,  occasional 
attenders  (less  than  three  times  per  week  but  two  or  more  times  per  month),  and  regular 
attenders  (at  least  three  times  per  week). 


e.  What  did  they  find  out? 

The  company  maintained  a work-force  of  approximately  5,000  employees  during  the  year,  with 
a turnover  rate  of  about  15  % . During  the  year  of  the  program,  778  employees  left  the  company, 
and  727  employees  joined  company.  Therefore,  there  were  5,727  employees  who  had  access 
to  the  fitness  program.  Of  this  number,  859  (15  %)  attended  the  program  at  least  twice  a month. 

Only  employees  who  participated  in  the  baseline  assessment  were  part  of  the  selection  process 
for  eligibility  to  attend  the  smoking  cessation  program.  That  is,  half  of  the  929  heavy  smokers 
(i.e.,  464)  and  half  of  the  492  light  smokers  (i.e.,  246)  were  selected  as  eligible  for  the 
program.  Of  those  eligible,  271  (38%)  participated  in  at  least  one  session  of  the  smoking 
cessation  program.  This  included  180  heavy  smokers  and  91  light  smokers. 


/.  How  were  the  results  used? 

The  results  indicated  that  employees  did  use  the  health  promotion  program,  and  suggested  that 
some  changes  in  employee  health  might  be  expected  as  a result  of  the  program.  Therefore,  the 
company  proceeded  with  outcome  evaluation. 


Methodological,  measurement,  and  ethical  issues: 

Ethical.  While  employee  identification  numbers  identified  those  who  used  the  fitness 
facility  and  how  frequently  they  used  it,  the  work-force  health  co-ordinator  had  no  way  of 
connecting  the  numbers  with  the  names  of  the  employees,  thus  maintaining  the  anonymity 
of  the  fitness  program  users. 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  heavy 


Part  4.  Outcome  Evaluation 


What  proportion  employees  of  tHe  company  quit  or  reducM 
smoking? 

What  proportion  of  employees  improved  their  fitness  level? 

What  proportion  of  employees  reduced  blood  pressure  from 
hypertensive  classification  (i.e.,  greater  than  140/90)  to  non- 
hypertensive? 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 

When  the  company  implemented  the  health  promotion  program,  it  was  done  on  a trial  basis. 
Part  of  implementing  the  program,  included  a 12-month  follow-up  to  assess  any  changes  among 
employees  compared  to  baseline  levels.  Employees  were  surveyed  before  the  implementation 
of  the  program  (see  Part  2)  and  again  one  year  later  to  measure  smoking  status,  fitness  level  and 
blood  pressure. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

The  five  company  nurses  spent  approximately  two  months  conducting  follow-up  assessments  of 
employees  and  documenting  the  information  along  with  an  identifier  code  (last  four  digits  of 
employee’s  phone  number  and  day  and  year  of  birth)  on  a summary  form.  The  forms  were  sent 
to  a keypunching  firm  for  data  entry  and  returned  to  the  work-force  health  co-ordinator  on  disk. 
The  co-ordinator  spent  20  hours  cleaning  the  data  of  errors,  matching  pre-test  and  post- test  data 
using  the  identifier,  and  analyzing  the  data.  In  order  to  complete  the  analyses  for  this 
evaluation,  a computer  and  software  capable  of  computing  simple  statistics  were  required. 
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c.  How  were  the  data  collected? 

The  assessment  described  in  Part  1 was  repeated  12  months  later.  Responses  from  employees 
at  follow-up  were  matched  with  their  responses  at  initial  assessment  by  matching  the 
identification  codes. 


d.  How  were  the  data  analyzed? 

Proportions  of  employees  in  each  category  of  fitness  level,  and  blood  pressure  were  compared 
at  pre-  and  post-test.  Only  those  employees  who  participated  both  in  the  original  assessment  and 
the  12-month  assessment  were  included  in  the  analyses.  Since  the  smoking  program  was  limited 
to  only  half  of  the  employees  who  smoked,  pre-  and  post-  comparisons  are  made  only  for  those 
who  were  selected  as  eligible  for  the  smoking  program. 


e.  What  did  they  find  out? 

Of  the  original  5,000  employees,  4,700  (94%)  participated  in  the  baseline  assessment.  The 
company  has  a turnover  of  about  15%  per  year.  Therefore,  when  post-testing  was  done  one 
year  later,  705  (15%  of  the  sample)  had  left  the  company.  Also,  188  (4%)  of  the  employees 
who  participated  in  the  original  assessment  did  not  participate  in  the  follow-up  assessment.  The 
final  sample  for  analysis  consisted  of  3,807  employees  (81%  of  the  original  sample). 

As  shown  in  Table  1 , there  were  fewer  heavy  smokers  at  the  end  of  the  program  among  eligible 
employees.  Among  all  employees  involved  in  the  pre-  and  post-testing,  the  proportion  of  fit  or 
very  fit  employees  increased,  and  a higher  proportion  of  employees  showed  non-hypertensive 
levels  of  blood  pressure  (i.e.,  non-problem  levels)  as  opposed  to  hypertensive  levels  (i.e., 
problem  levels)  after  the  implementation  of  the  programs. 

Table  2 shows  the  change  in  smoking  status  among  employees  eligible  for  the  smoking  cessation 
program.  As  shown  in  this  table,  20%  of  eligible  light  smokers  quit  smoking,  and  over  30% 
of  eligible  heavy  smokers  either  quit  (11%)  or  reduced  smoking  (21%). 

Changes  in  fitness  level  from  pre-  to  post-test  are  shown  in  Table  3.  Although  the  improvement 
in  fitness  level  across  all  employees  was  not  large,  there  was  a general  trend  for  overall 
improvement,  with  those  whose  were  already  fit  tending  to  stay  at  this  level,  while  those  who 
were  less  fit  tended  to  improve  overall. 
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Table  1.  Comparisons  of  Pre-  and  Post-test  Measures  (Those  Employees  Who  Did  Not 
Participate  in  Both  Measures  Were  Excluded) 


Pre-test 

Post-test 

Smoking  status*: 

former  smoker  (quit  smoking  within  the  last  12  months) 

- 

78 

(14%) 

light  smoker  (less  than  10  cigarettes  per  day) 

191 

(33%) 

228 

(40%) 

heavy  smoker  (10  or  more  cigarettes  per  day) 

381 

(67%) 

266 

(47%) 

Fitness  level: 

very  fit 

190 

(5%) 

245 

(6%) 

adequate  fitness 

1,485 

(39%) 

1,625 

(43%) 

less  than  adequate  fitness 

2,132 

(56%) 

1,937 

(51%) 

Blood  pressure  (tested  on  at  least  two  occasions): 

non-hypertensive  (BP  =140/90  or  less) 

3,046 

(80%) 

3,274 

(86%) 

hypertensive  (BP  with  higher  number  greater  than  140 
and/or  lower  number  greater  than  90,  tested  on  at  least  two 
occasions) 

761 

(20%) 

533 

(14%) 

^Analyzed  only  for  employees  selected  as  eligible  for  the  smoking  program. 
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Table  2.  Change  in  Smoking  Status  from  Pre-  to  Post-test  by  Original  Smoking 
Category 


Smoking  status  at  post-test 

Quit 

Smoking 

Light 

smoker 

Heavy 

smoker 

Smoking  status  at  pre-test: 

light  smoker  (less  than  10  cigarettes  per  day) 
(n=191) 

38 

(20%) 

150 

(79%) 

3 

(2%) 

heavy  smoker  (10  or  more  cigarettes  per  day) 
(n=381) 

40 

(10%) 

78 

(20%) 

263 

(69%) 

Table  3. 


Change  in  Fitness  Level  from  Pre-  to  Post-test  by  Original  Fitness  Level 


Fitness  level  at  post-test 

Very  fit 

Adequate 

fitness 

Less  than 
adequate 
fitness 

Fitness  level  at  pre-test: 

very  fit 

144 

34 

12 

(n=190) 

(76%) 

(18%) 

(6%) 

adequately  fit 

90 

1,353 

42 

(n=  1,485) 

(6%) 

(91%) 

(3%) 

less  than  adequately  fit 

11 

238 

1,883 

(n=2,132) 

(1%) 

(11%) 

(88%) 
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/.  How  were  the  results  used? 

Overall,  employee  health  levels  appeared  to  improve  modestly  over  the  12-month  period. 

Gains  were  apparent  for  all  three  areas  of  smoking  reduction,  improved  fitness  and  reduced 
prevalence  of  hypertension.  In  order  to  assess  the  extent  to  which  these  changes  appeared  to  be 
related  to  participation  in  the  program,  further  analyses  were  undertaken  evaluating  the 
relationships  between  (a)  participating  in  the  smoking  cessation  program  and  quitting  smoking, 
and  (b)  attending  the  fitness  program  and  improving  fitness  level. 


Methodological,  measurement  and  ethical  issues: 

Methodological.  This  evaluation  did  not  use  a control  group  to  assess  whether  the  changes 
that  occurred  were  actually  a result  of  the  workplace  health  program.  It  is  possible  that 
these  levels  of  improvement  would  have  occurred  without  the  program.  Thus,  the  findings 
provide  some  support  for  the  program’s  impact  but  do  not  provide  conclusive  evidence. 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  heavy 


Part  5.  Program  Delivery  by  Outcome  Evaluation 


How  many  employees  who  participated  m smoking  cessation 
program  quit  smoking  compared  to  employees  who  were  eligible 
the  program  but  did  not  participate?^^  ^^^^^^  ^^^^  ^ ^ ^ ^ 

What  proportion  of  employees  who  reported  attending  the  fitness 
program  an  average  of  two  times  per  month  or  more  improved  their 
level  of  fitness  compared  to  employees  who  did  not  attend  the  fitness 
program  m attended  at  less  Ikon  tMs  rate? 


a.  Who  was  asking  the  questions  and  why  did  they  want  the  information? 

The  previous  parts  of  the  evaluation  had  found  that  fewer  employees  were  heavy  smokers, 
fitness  had  increased,  and  the  proportion  of  employees  with  hypertensive  blood  pressure  levels 
had  decreased  from  the  year  before.  The  company  now  wanted  the  work-force  health  co- 
ordinator to  assess  the  relationship  between  improvement  and  participation  in  the  company’s 
programs. 

All  information  required  for  this  part  of  the  evaluation  had  previously  been  collected,  and  time 
1 (pre-test)  responses  were  linked  to  time  2 (post-test)  responses  by  means  of  an  eight  digit 
identification  code  (see  description  of  code  in  Part  2). 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

All  information  required  for  this  part  of  the  evaluation  had  previously  been  collected  and  entered 
in  the  computer.  The  work-force  health  co-ordinator  spent  approximately  two  hours  analyzing 
the  information.  In  order  to  complete  the  analyses  for  this  evaluation,  a computer  and  software 
capable  of  computing  simple  statistics  were  required. 


Workplace  Health  Promotion  Program 


88 


c.  How  were  the  data  collected? 

All  information  required  for  this  part  of  the  evaluation  had  previously  been  collected  during  the 
initial  and  follow-up  surveys.  In  addition  to  repeating  measures  of  smoking  status,  fitness  status, 
and  blood  pressure,  the  follow-up  summary  included  information  about  eligibility  for  the 
smoking  program,  and  attendance  at  the  smoking  cessation  program  and/or  the  fitness  program. 
This  information  was  keypunched  and  entered  into  the  computer  database. 


d.  How  were  the  data  analyzed? 

Change  in  smoking  status  was  assessed,  comparing  those  who  participated  in  the  smoking 
cessation  program  with  those  who  were  eligible  for  the  program  but  did  not  participate. 
Similarly,  change  in  fitness  level  was  compared  for  those  who  attended  the  fitness  program  and 
those  who  did  not.  Employees  were  categorized  as  attending  the  fitness  program  if  they  reported 
attending  twice  a month  or  more. 


e.  What  did  they  find  out? 

As  described  in  Part  4,  3,807  employees  (81%  of  the  original  sample)  participated  in  both  the 
initial  assessment  and  the  follow-up.  Of  these,  1,136  were  either  light  or  heavy  smokers  at  the 
time  of  the  pre-test,  and  572  of  these  smokers  were  offered  the  smoking  cessation  program. 
Table  1 shows  the  pre-  and  post-test  smoking  status  for  the  215  who  participated  in  the  smoking 
cessation  program  compared  to  the  357  smokers  who  were  eligible  for  the  program  but  did  not 
participate.  These  analyses  indicated  that  employees  who  participated  in  the  smoking  cessation 
program  reduced  or  quit  smoking  at  a greater  rate  than  did  those  who  were  eligible  but  did  not 
participate. 

Table  2 shows  change  in  smoking  status  for  all  employees  selected  as  eligible  for  the  smoking 
cessation  program  (including  those  who  participated  in  the  program  as  well  as  those  who  did  not) 
compared  to  those  who  were  not  selected  as  eligible  for  the  program.  As  is  evident  from  this 
table,  the  group  selected  as  eligible  for  the  smoking  cessation  program  showed  a greater 
reduction  in  smoking  than  occurred  among  those  who  were  not  eligible,  suggesting  an  impact 
of  the  smoking  cessation  program. 

Table  3 shows  the  pre-  and  post-test  fitness  levels  of  the  3,239  (85%)  employees  who  were 
classified  as  not  attending  attend  the  fitness  program  (i.e.,  never  attended  or  attended  less  than 
twice  a month)  compared  to  the  568  (15%)  who  attended  the  fitness  program  at  least  twice  a 
month.  As  these  data  indicate,  a greater  proportion  of  employees  who  attended  the  fitness 
program  on  a regular  basis  achieved  or  maintained  at  least  an  adequate  level  of  fitness  than  did 
employees  who  did  not. 
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Table  1.  Change  in  Smoking  Status  of  Light  and  Heavy  Smokers  Who  Participated  in 
the  Smoking  Cessation  Program  Compared  to  Smokers  Who  Were  Offered 
the  Program  but  Did  Not  Participate 


Smoking  status  at 

post-test 

Smoking  status  at  pre-test 

Quit 

smoking 

Light 

smoker 

Heavy 

smoker 

Those  who  participated  in  the  smoking  cessation  program 
(n=215): 

light  smoker  (n=72) 

32 

(44%) 

40 

(56%) 

0 

(0%) 

heavy  smoker  (n=  143) 

36 

(25%) 

58 

(41%) 

49 

(34%) 

Those  who  were  offered  access  but  did  not  participate  in 
the  smoking  cessation  program  (n=357): 

light  smoker  (n=  1 19) 

6 

(5%) 

110 

(92%) 

3 

(3%) 

heavy  smoker  (n=238) 

4 

(2%) 

20 

(8%) 

214 

(90%) 

Note:  These  figures  include  only  those  smokers  who  participated  in  both  the  pre-test  and  the 
post-test. 
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Table  2.  Change  in  Smoking  Status  of  Light  and  Heavy  Smokers  Eligible  for  the 
Smoking  Cessation  Program  Compared  to  Smokers  Who  Were  Not  Eligible 


Smoking  status  at  post-test 

Smoking  status  at  pre-test 

Quit 

smoking 

Light 

smoker 

Heavy 

smoker 

All  employees  selected  as  eligible  for  the  smoking 
cessation  program  (n=572): 

light  smoker  (n  = 191) 

38 

(20%) 

150 

(79%) 

3 

(2%) 

heavy  smoker  (n=381) 

40 

(10%) 

78 

(20%) 

263 

(69%) 

Employees  who  were  not  selected  as  eligible  for  the 
smoking  cessation  program  (n=564): 

light  smoker  (n  = 170) 

12 

(7%) 

155 

(91%) 

3 

(2%) 

heavy  smoker  (n=394) 

9 

(2%) 

43 

(11%) 

342 

(87%) 

Note:  These  figures  include  only  those  smokers  who  participated  in  both  the  pre-test  and  the 
post-test. 
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Table  3.  Change  in  Fitness  Level  for  Those  Who  Attended  the  Fitness  Program 
Compared  to  Those  Who  Did  Not 


Fitness  level  at  post  test 

Very  fit 

Adequate 

fitness 

Less  than 
adequate 
fitness 

Fitness  level  at  pre-test: 

Those  who  took  the  fitness  program  at  least 
twice  per  month  (n=568): 

very  fit  (n=42) 

41 

0 

1 

(98%) 

(0%) 

(2%) 

adequate  fit  (n=371) 

71 

292 

8 

(19%) 

(79%) 

(2%) 

less  than  adequate  (n  = 155) 

2 

97 

56 

(1%) 

(63%) 

(36%) 

Those  who  did  not  take  the  fitness  program 
(n=3,239): 

very  fit  (n  = 148) 

103 

34 

11 

(70%) 

(23%) 

(7%) 

adequate  fit  (n  = 1 , 1 14) 

19 

1,061 

34 

(2%) 

(95%) 

(3%) 

less  than  adequate  (n  = 1,977) 

9 

141 

1,827 

(<1%) 

(7%) 

(92%) 
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/.  How  were  the  results  used? 

Overall,  employees  who  took  part  in  the  smoking  cessation  program  were  more  likely  to  quit 
or  reduce  smoking,  and  those  who  attended  the  fitness  program  were  more  likely  to  improve 
their  fitness  than  non-attenders.  Therefore,  the  company  has  decided  to  continue  the  health 
promotion  program. 

Since  part  of  the  evaluation  involved  random  selection  of  employees  who  were  eligible  for  the 
smoking  cessation  program,  those  who  were  not  selected  provided  a control  group  for  further 
analyses.  Therefore,  cost-effectiveness  analyses  of  the  smoking  program  were  done  in  Part  6 
of  the  evaluation.  The  fitness  program  involved  no  control  group.  Therefore,  although  the 
results  of  the  program  were  positive,  cost-effectiveness  analyses  were  not  justified.  The  extent 
that  the  fitness  program  caused  the  change  could  not  be  ascertained  from  the  evaluation  as 
designed  (see  Methodological  note). 


Methodological,  measurement  and  ethical  issues: 

Methodological.  As  was  clear  in  Table  3,  those  who  attended  the  fitness  program  were 
more  likely  to  improve  in  fitness  level  than  those  who  did  not  attend.  Although  the 
company  felt  these  results  were  sufficient  to  justify  maintaining  the  program,  it  should  be 
recognized  that  these  results  do  not  necessarily  demonstrate  the  effectiveness  of  the 
program.  Since  there  was  no  control  or  comparison  group,  it  is  impossible  to  know  from 
the  present  data  how  many  people  who  attended  the  fitness  program  would  have  attended  a 
fitness  program  elsewhere  if  the  company  program  did  not  exist.  In  other  words,  when 
attendance  at  a program  is  based  on  self-selection,  a control  group  is  needed  to  determine 
to  what  extent  change  is  caused  by  people  deciding  to  change  (regardless  of  the  program), 
and  to  what  extent  the  program  contributes  to  the  change.  Although  a control  condition 
was  not  used  for  the  fitness  program,  from  the  company’s  perspective,  it  seemed 
worthwhile  to  provide  the  program  as  a vehicle  for  improved  fitness,  regardless  of  the 
extent  to  which  improvements  could  be  directly  attributable  to  the  program  (as  opposed  to 
self-selection). 

A control  group  was  available  for  evaluating  the  smoking  cessation  program,  namely  the 
group  of  smokers  who  were  not  randomly  selected  as  eligible  for  the  program.  If  change 
in  smoking  among  those  who  attended  the  company’s  program  was  caused  solely  by  self- 
selection (that  is,  those  who  attended  the  program  would  have  attended  some  other  program 
and/or  would  have  quit  anyway),  then  the  rate  of  quitting  would  have  been  the  same  for 
smokers  eligible  for  the  program  as  for  those  not  eligible.  Since  the  rate  among  those 
eligible  was  higher,  this  suggests  that  some  change  was  due  to  the  program. 
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Level  of  difficulty:  difficult 
Level  of  resources  needed:  heavy 


Part  6.  Economic  Evaluation 


What  was  the  cost  extra  person  quitting 

smoking  as  a result  of  the  smoking  cessation  program? 


a.  Who  was  asking  the  questions  and  why  did  they  want  the  information? 

The  previous  parts  of  the  evaluation  had  found  that  a larger  proportion  of  pre-test  smokers  who 
participated  in  the  program  had  quit  smoking,  compared  with  pre-test  smokers  who  were  offered 
access  to  the  program  but  did  not  participate.  However,  since  some  smokers  quit  without 
participating  in  the  program,  the  company  wanted  to  determine  how  many  additional  smokers 
quit  as  a result  of  the  program,  and  what  it  cost  the  company  for  each  of  these  extra  quitters. 
This  information  was  needed  to  help  the  company  decide  whether  to  make  the  program  available 
to  the  other  half  of  the  smoking  employees  who  had  not  been  selected  in  the  initial  phase. 

Information  had  already  been  collected  on  the  pre-  and  post-test  smoking  status  of  all  employees. 
However,  it  was  necessary  to  estimate  the  costs  to  the  company  of  providing  the  smoking 
program. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

As  the  required  information  concerning  the  quitters  in  both  the  selected  and  unselected  smokers 
groups  had  been  collected  and  entered  in  the  computer,  only  cost  information  remained  to  be 
collected.  In  order  to  estimate  program  costs  the  work-force  health  co-ordinator  obtained  from 
the  accounting  department  the  total  billings  from  the  fee-for-service  program  leader,  and  from 
the  human  resources  department  the  company’s  average  cost  per  employee- working  hour  (after 
adjustment  of  the  normal  cost  per  employee-hour  for  statutory  holidays,  vacation,  paid  sick  time, 
other  benefits,  paid  breaks,  and  staff  training).  The  work-force  health  co-ordinator  also  obtained 
from  the  smoking  program  leader  a record  of  the  total  number  of  employee  sessions  provided 
(i.e.,  the  sum  for  all  sessions  of  the  number  of  employees  participating). 
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c.  How  were  the  data  collected? 

All  information  required  for  pre-  and  post-test  smoking  status  had  been  collected  previously 
during  the  initial  and  follow-up  surveys;  this  included  information  on  whether  or  not  the 
employee  had  participated  in  at  least  one  session  of  the  program.  This  information  had  also 
been  keypunched  and  entered  into  the  computer  database. 

The  total  number  of  employee-sessions  provided  by  the  smoking  program  was  collected  by  the 
program  leader.  The  adjusted  employee  cost  per  hour  was  calculated  by  the  human  resources 
department  from  the  company’s  payroll  expenditure  reports.  The  fee-for- service  billings  were 
taken  directly  from  the  accounting  department’s  general  ledger. 


d.  How  were  the  data  analyzed? 

The  difference  in  proportion  of  quitters  was  calculated  for  the  smoker  groups  selected  as  eligible 
and  not  eligible  for  the  smoking  cessation  program. 

The  cost  of  the  smoking  program  to  the  company  was  estimated  as  the  sum  of  the  external 
program  leader’s  fee-for- service  billings  plus  the  cost  of  the  30  minutes  of  employee  time  for 
each  occasion  an  employee  attended  a program  session.  The  employee  time  cost  was  calculated 
as  the  product  of  the  total  number  of  employee-sessions  provided  x 0.5  hrs  x the  estimated 
adjusted  employee  cost  per  hour.  The  cost  of  program  materials  was  included  in  the  fee-for- 
service  billings,  and  zero  cost  was  assigned  for  space  because  the  room  used  for  the  program 
sessions  would  have  otherwise  remained  unused. 

The  mean  cost  to  the  company  per  employee  with  access  to  the  program  was  calculated  by 
dividing  program  cost  by  the  number  of  employees  in  the  group  offered  access. 

The  cost  to  the  company  per  extra  quitter  achieved  by  the  program  was  estimated  by  dividing 
the  mean  cost  per  employee  with  access  to  the  program  by  the  difference  in  proportions  of 
quitters  in  the  groups  with  and  without  access. 
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e.  What  did  they  find  out? 

As  described  in  Parts  4 and  5,  78  of  the  572  smokers  who  were  eligible  for  the  smoking 
cessation  program  and  completed  both  pre-  and  post-test  data  collections  had  quit  smoking  at 
post-test,  i.e.,  the  proportion  of  quitters  was  0.14  (see  Table  1.)  As  shown  in  Table  1,  the 
proportion  of  post-test  quitters  in  the  group  not  eligible  for  the  smoking  program  was  21  of  564 
or  0.04.  Therefore,  the  difference  between  the  groups  in  proportion  of  quitters  was  0.10. 

The  costs  to  the  company  of  the  smoking  cessation  program  are  shown  in  Table  2.  The  total 
cost  of  making  the  program  accessible  to  the  710  employees  who  were  originally  selected  as 
eligible  for  the  program  was  $68,115  (i.e.,  a mean  cost  per  selected  employee  of  $95.94).  It 
is  assumed  that  zero  cost  was  incurred  for  the  unselected  group. 

Accordingly,  the  cost  per  extra  quitter  achieved  by  the  smoking  program  was  estimated  at 
$95.94  ^ 0.10  = $959.40. 


Table  1.  Quitting  Rate  of  Employees  Selected  as  Eligible  for  the  Smoking  Cessation 
Program  Compared  to  Employees  Who  Had  Not  been  Selected 


Employees  eligible 
for  smoking 
cessation  program 

Employees  who  were 
not  eligible  for  smoking 
cessation  program 

number  of  smokers  who  quit  smoking 

78 

21 

number  of  smokers  who  continued  smoking 

494 

543 

proportion  of  smokers  who  quit 

0.14 

0.04 

Note:  These  figures  include  only  those  smokers  who  participated  in  both  the  pre-test  and  post-test. 
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Table  2.  Cost  to  the  Company  of  the  Smoking  Program 


Cost  Element 

Cdst 

Fee-for-service  billings 

$35,000.00 

Employee  time  cost: 

2,236  employee-sessions  x 0.5  x $29. 62/hr 

$33,115.00 

Total  Cost 

$68,115.00 

Mean  cost  per  employee  originally  selected  as 
eligible  for  the  program  (n=710) 

$95.94 

Estimated  cost  per  extra  quitter  achieved  by  the 
smoking  program:  $95.94  -r  0.10 

$959.40 

(f)  How  were  the  results  used? 

Although  the  program  was  successful  in  helping  employees  quit  smoking,  the  cost  to  the 
company  was  fairly  high.  To  reduce  costs  but  still  maintain  the  program,  the  company  decided 
to  offer  the  smoking  cessation  program  outside  of  work  hours  (thus  saving  employee  time  costs). 
Since  this  change  might  well  reduce  the  success  rate  of  the  program,  the  company  would 
continue  monitoring  the  program  for  another  year  (offering  the  program  to  only  half  of  the 
employees). 
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Methodological,  measurement  and  ethical  issues: 

Measurement.  In  this  example,  costs  were  calculated  only  from  the  company’s  perspective 
and  only  using  basic  cost  estimates.  In  particular,  benefits  are  not  calculated  as  potential 
cost-offsets  for  the  company.  For  example,  employee  time  costs  are  based  on  time  away 
from  the  job  to  attend  the  smoking  cessation  program.  However,  if  quitting  or  reducing 
smoking  resulted  in  fewer  sick  days,  the  net  time  away  from  the  job  for  those  who  attended 
the  program  might  be  somewhat  less  (and,  correspondingly,  the  costs  to  the  company  for 
the  program  would  be  less). 

While  this  example  focused  only  on  the  costs  to  the  company  of  the  program,  a broader 
economic  evaluation  might  take  into  consideration  other  costs  and  benefits,  such  as  the  cost 
of  buying  cigarettes  by  employees  who  continue  to  smoke,  reduced  heath  care  costs  for 
those  who  quit  smoking,  losses  in  taxes  from  employees  who  quit  smoking,  and  so  on. 
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Note:  This  example  involves  the  collection  and  analyses  of  a lot  of  data.  These  analyses 
are  not  reported  in  detail  as  this  type  of  project  should  only  be  undertaken  by  persons  or 
groups  who  have  access  to  design  and  analysis  expertise.  The  example  is  provided  to  show 
the  general  approach  to  community  level  needs  assessment  for  health  promotion 
programming. 


Program:  Within  the  field  of  health  care,  there  has  been  a shift  in  emphasis  from  the 
traditional  treatment  and/or  rehabilitation  oriented  approach  to  one  of  health  promotion  and 
prevention.  Health  care  providers,  public  and  private  funding  organizations  and  research  driven 
investigations  are  giving  more  attention  to  health  promotion  as  an  important  aspect  of  overall 
health  care  provision.  In  Ontario,  the  Premier’s  Council  on  Health  Strategy  adopted  the  World 
Health  Organization’s  definition  of  health  as  not  only  the  absence  of  disease,  but  also  as  a 
positive  resource  that  allows  people  to  cope  with,  adapt  to,  and  influence  the  pressures  of  daily 
living.  This  council  has  also  identified  health  promotion  as  a major  goal. 

Health  promotion  focuses  on  the  development  of  good  health  and  healthy  lifestyles  rather  than 
health  problems.  It  is  meant  to  develop,  maintain  and  enhance  levels  of  health,  and  at  the  same 
time,  reduce  health  problems.  Important  factors  contributing  to  the  process  of  health  promotion 
and  healthy  lifestyles  are  personal  responsibility,  structural  arrangements  in  society,  economic 
conditions,  and  socio-cultural  practices.  When  these  are  related  specifically  to  substance  abuse, 
the  dynamic  interactions  between  the  drug,  the  person  and  the  environment  influence  the 
reduction  of  substance  abuse  problems  and  the  promotion  of  health  in  society. 

Corresponding  to  the  shift  in  emphasis  from  a treatment  orientation  to  a health  promotion 
orientation,  local  communities  are  being  recognized  as  major  influences  and  important 
contributors  to  the  promotion  of  better  health  and  healthy  lifestyles. 

In  order  to  plan  health  promotion  programs,  systematic  data  collection  regarding  the 
characteristics  of  the  community  is  useful.  This  involves  addressing  the  following  kinds  of 
questions.  Who  lives  there?  What  do  they  do?  What  are  their  perceptions  of  community  needs 
and  present  services?  Which  health-related  areas  are  most  in  need  of  attention?  What  is  the 
social,  cultural,  economic  and  demographic  make-up  of  the  community?  Are  there  important 
questions  that  must  be  addressed  prior  to  any  actual  program  planning  and  implementation?  In 
other  words,  a needs  assessment  is  necessary  to  learn  the  facts  and  opinions  about  the 
community’s  current  health  status  before  taking  action. 
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When  possible,  it  is  better  to  use  several  different  sources  of  data  in  a needs  assessment  to 
ensure  greater  validity.  The  community-based  needs  assessment  used  in  the  present  example 
shows  one  approach.  Taken  together  with  other  information  sources  such  as  census  data,  social 
indicators,  service  utilization  and  formal  interviews,  the  community  needs  assessment  helps 
provide  a clear  overview  of  the  community  which  can  be  used  to  guide  the  development  of 
health  promotion  programs.  The  results  of  the  survey  provide  the  health  promotion  planner  with 
information  regarding  the  range  of  local  attitudes,  norms,  and  perspectives  on  alcohol  and  other 
drug  problems,  as  well  as  on  social  problems  in  general.  In  addition,  the  survey  can  measure 
the  respondent’s  awareness  of  what  available  resources  are  in  the  community. 

Ideally,  the  community  needs  assessment  provides  preliminary  insights  into  the  perceived 
characteristics  of  alcohol/drug  users,  particularly  those  users  who  are  at  risk  for  current  or  future 
problems.  These  assessments  should  also  point  to  the  individual,  group  and  community-level 
factors  that  are  important  to  the  process  of  health  promotion  in  the  community,  and  show  the 
extent  to  which  these  various  factors  are  interrelated.  At  the  broadest  level,  the  needs 
assessment  should  provide  a reflection  of  local  environments  and  institutional  constraints  that 
must  be  taken  into  account  in  achieving  health  promotion  goals. 

The  present  needs  assessment  involved  face-to-face  interviews  with  over  200  community 
residents.  It  was  undertaken  as  the  first  step  in  program  planning,  to  help  build  appropriate 
policies  and  programs  for  alcohol  and  drug-related  issues.  The  needs  assessment  was  also  to 
be  used  to  provide  information  that  would  help  promote  the  accessibility  of  the  resources 
available  in  the  community,  including  new  services,  existing  services,  and  informal  networks. 

This  example  includes  quality  enhancement  questions  at  one  level  of  evaluation: 

Part  1.  Needs  Assessment. 


Community  Needs  Assessment  for  Health  Promotion 


100 


Level  of  difficulty:  difficult 
Level  of  resources  needed:  heavy 


Part  1.  Needs  Assessment 


From  the  perspective  of  the  community: 

What  are  the  major  problems  facing  the  community  with  respect  to 
alcohol  and  drugs? 

Who  should  be  the  primary  targets  of  health  promotion  programs  directed 
at  alcohol  and  drug  use? 

What  new  health  promotion  programs  should  be  developed  and/or 
implemented  to  prevent  problems  related  to  alcohol  and/or  drug^  m 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 

The  District  Health  Council  conducted  the  needs  assessment  in  order  to  obtain  current 
information  regarding  the  community.  This  information  was  to  be  used  directly  in  planning  new 
health  promotion  programs  for  preventing  problems  related  to  alcohol  and/or  drug  use. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

Human  resources  were  required  to  conduct  face-to-face  interviews  with  200  randomly  chosen 
representative  community  members  and  20  key  informants  who  worked  in  the  community.  Ten 
community  members  were  hired,  trained  and  provided  with  a list  of  addresses  to  visit. 
Additional  resources  were  required  to  print  copies  of  the  questionnaire.  A staff  member  of  the 
District  Health  Council  spent  nine  hours  a week  for  three  months  training  and  co-ordinating  the 
interviewers,  organizing  data  entry,  and  analyzing  the  data.  A computer  and  software  were 
required  for  analyzing  the  data. 
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c.  How  were  the  data  collected? 

Ten  community  members  conducted  face-to-face  structured  survey  interviews  with  220 
respondents,  including  20  key  informants.  The  completed  surveys  were  returned  to  the  District 
Health  Council. 


d.  How  were  the  data  analyzed? 

The  completed  surveys  were  coded  and  the  information  was  entered  on  the  computer  and 
analyzed  using  a statistical  software  package.  Frequency  distributions  and  correlations  were 
computed  to  identify  patterns  that  could  serve  as  the  basis  for  recommendations. 


e.  What  did  they  find  out? 

Respondents  expressed  numerous  positive  feelings  and  perceptions  about  their  community.  They 
felt  the  community  was  safe,  that  members  cared  about  the  community,  that  there  were 
numerous  social  and  recreational  activities  available,  and  that  their  immediate  neighbourhoods 
were  clean  and  safe  for  children.  However,  they  also  felt  that  there  was  a great  deal  of  apathy 
and  a lack  of  strong  community  leadership.  They  indicated  that  they  would  like  to  see  a more 
active  police  presence  in  the  community,  particularly  with  respect  to  the  enforcement  of  alcohol 
and  drug  policies. 

Even  though  community  members  and  key  informants  felt  that  drug  use  in  the  community  was 
a problem,  they  seemed  to  place  a greater  emphasis  on  alcohol  use.  Although  respondents  felt 
that  most  groups  were  appropriate  targets  for  interventions,  most  felt  that  youth  aged  12  to  18 
and  young  adults  should  be  the  primary  recipients  of  drug  and  alcohol  prevention  activities  and 
programs. 


/.  How  were  the  results  used? 

The  following  outlines  the  initial  plans  for  health  promotion  and  prevention  programs  that 
resulted  from  the  needs  assessment.  These  programs  follow  a model  that  focuses  on  strategies 
and  goals  for  health  promotion  activities  targeted  toward  the  person,  the  drug  and  the 
environment.  The  kinds  of  strategies  that  can  be  utilized  to  improve  the  health  status  of 
community  members  include  influence,  control,  competence  development,  and  environmental 
design.  The  goals  of  the  model  are  health  enhancement  (to  increase  the  person’s  desire  to 
abstain  and/or  control  their  drinking),  risk  avoidance  (to  maintain  the  person’s  low  level  of  use), 
and  risk  reduction  (to  reduce  the  person’s  use  before  health  or  social  problems  develop). 
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The  primary  group  that  was  targeted  were  young  people  between  12  and  18  years  of  age. 
Programs  that  were  recommended  for  development  and/or  implementation  included  the 
following: 

• providing  activities  that  would  interest  young  people  and  would  provide 
alternatives  to  alcohol  and  other  drug  use; 

• advocating  and  working  with  the  school  boards  in  the  community  to  develop  and 
implement  an  alcohol  and  drug  policy  that  would  include  the  components  of 
education,  early  identification  and  discipline;  and 

• providing  programs  for  families  that  would  serve  to  complement  and  reinforce  the 
messages  and  programs  young  people  were  being  exposed  to  outside  the  home. 


A secondary  target  was  the  community  in  general.  The  following  recommendations  were 
designed  to  provide  potential  solutions  to  the  problems  in  the  community  as  a whole  and  to  raise 
community  awareness  in  order  to  make  the  proposed  programs  more  acceptable: 

• publicize  the  existence  of  services  within  the  community; 

• advocate  with  the  municipal  politicians  to  develop  and  implement  a municipal 
alcohol  policy  that  governs  the  sale  and  distribution  of  alcohol  in  municipally 
owned  facilities;  and 

• implement  a server  intervention  program  in  licensed  facilities  to  ensure  that 
alcohol  is  served  responsibly  and  appropriately,  which  would  include  identifying 
and  not  serving  under-age  patrons. 


At  the  present  time  a number  of  steps  have  been  taken.  In  the  case  where  existing  programs  are 
available  (e.g.,  server  training,  school  policy),  work  has  begun  in  order  to  implement  these 
programs.  Other  recommendations  such  as  the  family  program  and  the  alternative  activities  for 
youth  are  being  discussed  and  reviewed.  It  is  anticipated  that  it  will  take  about  two  years  to 
implement  the  first  series  of  recommendations.  At  that  time,  another  needs  assessment  should 
be  conducted  to  determine  whether  changes  have  occurred  in  the  community,  whether  the 
programs  that  have  been  implemented  have  begun  to  have  an  effect,  and  what  new  programs 
should  be  considered. 
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Methodological,  measurement,  and  ethical  issues 

Methodological.  One  key  to  a valid  needs  assessment  is  the  selection  of  respondents  for 
the  survey.  In  order  to  ensure  that  the  sample  is  representative  of  the  entire  population,  a 
random  selection  procedure  was  implemented  using  the  voter  registration  list  as  the  sample 
pool. 

Measurement.  In  order  to  ensure  that  questions  on  the  survey  were  understandable  and 
acceptable  to  respondents,  the  instrument  was  pilot-tested  with  a variety  of  community 
representatives,  and  problematic  questions  were  revised  or  removed. 

Ethical.  Because  sensitive  questions  were  included  on  the  questionnaire,  it  was  important 
to  ensure  confidentiality.  No  names  were  recorded,  and  records  as  to  which  addresses 
were  approached  were  destroyed  after  the  representativeness  of  the  sample  was  determined. 
In  addition,  no  one  under  the  age  of  18  was  asked  to  respond,  and  all  respondents  were 
free  to  refuse  to  answer  a question  or  terminate  the  interview  at  any  point.  Finally, 
because  the  survey  was  intended  to  cover  the  entire  community,  and  interviewers  had  to 
enter  houses  to  conduct  the  interviews,  interviewers  were  potentially  at  risk.  Interviewers 
were  told  to  tell  respondents  about  the  intent  of  the  study,  and  that  they  were  part  of  a 
survey  team  in  the  area.  In  addition,  if  they  felt  at  all  ill-at-ease  about  doing  interviews  in 
a particular  neighbourhood,  they  were  told  to  go  in  teams  or  to  exclude  specific  houses 
where  they  felt  they  would  be  put  at  risk.  No  problems  regarding  the  safety  of 
interviewers  were  encountered. 
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F.  An  EAP  Counselling  Program 


Program:  The  program  is  an  off-site  service  available  to  1,074  employees.  Of  this  number, 
392  are  highly  skilled  staff.  In  supporting  the  EAP,  the  management  was  concerned  about 
addressing  substance  abuse  and  other  problems  among  skilled  employees  in  particular,  because 
these  employees  are  more  costly  to  train  and  more  difficult  to  replace  than  unskilled  employees. 

Employees  and  their  families  may  use  the  EAP  for  a variety  of  problems.  The  services  available 
are  individualized  short-term  counselling,  case  management,  crisis  intervention,  and  referral  for 
additional  services.  Some  employees  are  encouraged  to  attend  the  EAP  by  their  supervisors, 
while  others  use  the  services  on  their  own. 

The  program  was  initially  set  up  to  address  alcohol  or  drug-related  problems  in  the  workplace, 
but  currently  addresses  a variety  of  personal  and  work-related  problems  experienced  by 
employees. 

This  example  includes  quality  enhancement  questions  for  four  levels  of  evaluation: 


Part  1.  Client  Characteristics 


Part  2.  Program  Delivery 

Part  3.  Client  Satisfaction 


Part  4.  Client  Satisfaction  by  Program  Delivery. 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  moderate 


Part  1.  Client  Characteristics 


To  what  extent  wm  the  EAP  used  dy  skiUed  employees? 

What  proportion  of  employees  using  the  EAF  addressed  personal,  work- 
related,  and/or  substance  abuse  issues? 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 

The  EAP  was  developed  to  help  employees  address  problems  that  might  be  affecting  their  work. 
Since  skilled  employees  are  generally  more  valuable  to  the  company,  the  company  managers 
were  particulcirly  interested  in  determining  the  extent  to  which  the  EAP  was  being  used  by 
skilled  employees. 

Therefore,  the  first  consideration  of  this  evaluation  was  to  determine  the  percentage  of  skilled 
staff  who  used  the  EAP.  The  company  was  also  interested  in  knowing  the  nature  of  problems 
for  which  employees  sought  help.  Specifically,  they  wanted  to  determine  the  proportion  of 
employees  who  were  addressing  either  personal,  work-related,  or  substance  abuse  issues. 
Analyses  of  the  types  of  problems  reported  by  employees  attending  the  EAP  would  be  useful  for 
ensuring  that  EAP  counsellors  had  appropriate  training  for  addressing  the  most  frequent  problem 
areas. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

The  information  necessary  to  address  these  questions  was  collected  routinely  during  the  EAP 
users’  intake  interviews.  An  EAP  staff  person  spent  eight  hours  reviewing  case  files,  organizing 
and  analyzing  the  information,  and  writing  a report. 
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c.  How  were  the  data  collected? 

The  EAP  counsellor  conducted  a comprehensive  assessment  at  intake,  including  obtaining  the 
information  needed  for  the  present  analyses.  The  designation  of  skilled/unskilled  was  part  of 
the  employee’s  job  code  in  company  records.  During  assessment,  the  employee  and  the 
counsellor  together  identified  the  problems  to  be  addressed.  This  assessment  included 
identifying  the  most  important  problem  for  which  the  employee  was  using  the  EAP.  Only  the 
primary  problem  was  used  in  the  data  analysis. 


d.  How  were  the  data  analyzed? 

The  information  was  entered  in  the  computer  using  EPI  INFO  (see  Appendix  A).  The  percent 
of  skilled  versus  unskilled  employees  using  the  EAP  and  the  proportions  of  people  using  the 
EAP  for  various  reasons  was  calculated  overall  and  by  category  (i.e.,  skilled/unskilled)  of 
employee. 

To  illustrate  the  results,  graphs  were  prepared  using  Harvard  Graphics  (see  Appendix  C). 


e.  What  did  they  find  out? 

One  hundred  and  thirty-three  (12%)  employees  used  the  EAP  services  in  the  first  24  months 
of  the  contract.  Fifty-eight  (44%)  were  designated  skilled  and  75  (56%)  unskilled.  Since  there 
were  392  employees  designated  skilled  and  682  designated  unskilled  in  the  company,  these 
numbers  indicate  that  the  EAP  was  used  by  15%  of  all  skilled  employees  and  11%  of  all 
unskilled  employees.  A series  of  pie  charts  (prepared  using  Harvard  Graphics)  indicating  the 
percent  of  skilled  versus  unskilled  employees  using  the  EAP  is  shown  in  Figure  1 . 

As  shown  in  Figure  2 and  Table  1 , the  most  frequent  primary  problems  for  which  employees 
attended  the  EAP  were  work-related  issues  and  alcohol  and  drug  use  by  self.  Reasons  for  using 
the  EAP,  however,  differed  between  skilled  and  unskilled  employees.  Skilled  workers  tended 
to  be  more  likely  than  non-skilled  workers  to  address  personal  substance  abuse  problems.  Non- 
skilled  employees  tended  to  be  more  likely  to  seek  help  for  workplace  problems.  (See  Table 
1 for  detailed  results  and  Figure  3 for  a bar  graph  display.) 

Within  job-related  problems,  some  of  the  specific  issues  addressed  were:  job  stress,  sexual 
harassment,  conflict  with  supervisors,  conflict  with  co-workers,  and  too  much 
work/responsibilities.  "Other"  problems  included  emotional  health,  physical  health,  sexual 
assault,  and  legal  problems. 
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Figure  1 . Utilization  of  EAP  by  Skilled  and  Unskilled  Employees 


Skilled 
(392)  36% 


(682)  64% 

Use  of  EAP  by  unskilled  employees  Use  of  EAP  by  skilled  employees 


% skilled  and  unskilled  employees 


Table  1.  Primary  Problem  Addressed  by  Skilled  and  Unskilled  Employees  Who 
Attended  the  EAP 


Problem  addressed 

Skilled 

employees 

n=58 

Unskilled 

employees 

n=75 

All 

employees 

n-133 

alcohol/drug  use  - self 

22 

(38%) 

14 

(19%) 

36 

(27%) 

- other 

6 

(10%) 

7 

(9%) 

13 

(10%) 

work-related  issues 

15 

(26%) 

28 

(37%) 

43 

(32%) 

personal  problems  - family 

9 

(16%) 

11 

(15%) 

20 

(15%) 

- financial 

3 

(5%) 

9 

(12%) 

12 

(9%) 

other 

3 

(5%) 

6 

(8%) 

9 

(7%) 
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Figure  2.  Primary  Problem  Addressed  by  Employees  Who 

Attended  the  EAP 


Alcohol/drug  use  other 
(13)  10% 


Job  related  issues 
(43)  32% 


Alcohol/drug  use  self 
(36)  27% 


Other 
(9)  7% 


Personal  - financial 
Personal  - family  0^)  9% 

(20)  15% 


Figure  3.  Primary  Problem  Addressed  by  Skilled  and  Unskilled 
Employees  Who  Attended  the  EAP 

% of  employees  addressing  the  issue 
Issues  addressed  o%  10%  20%  30%  40%  50% 

Alcohol/drug  use  self 

Alcohol/drug  use  other 

Job  related  issues 
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This  part  of  the  evaluation  addressed  two  main  concerns:  assessing  the  use  of  the  EAP  by 
skilled  employees  and  identifying  the  nature  of  the  problems  addressed  by  the  EAP  in  order  to 
plan  future  services. 

The  results  demonstrated  that  skilled  employees  used  the  EAP  at  least  as  often  as  unskilled 
employees.  These  findings  showed  that  the  EAP  was  meeting  one  of  its  main  goals. 

The  most  prevalent  problems  for  which  employees  came  to  the  EAP  were  substance  abuse  and 
job-related  issues.  In  view  of  this,  the  company  is  working  with  the  EAP  to  develop  an  in-house 
"Workplace  Wellness"  program  which  focuses  on  low-risk  drinking,  substance  abuse  prevention, 
and  job-related  stress. 

Finally,  while  addressing  the  two  quality  enhancement  questions,  the  company  managers  also 
observed  that  the  high  rate  of  overall  use  of  the  EAP  (12%  over  two  years)  suggested  that  the 
program  was  both  acceptable  and  accessible  to  the  employees. 
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Level  of  difficulty:  easy 

Level  of  resources  needed:  minimal 


Part  2.  Program  Delivery 


What  were  the  main  presenting  problems  for  which  referrals  were  made 
to  supplement  EAF  services? 


a.  Who  was  asking  the  question  and  why  did  they  want  this  information? 

The  EAP  was  set  up  to  provide  short-term  counselling,  case  management,  crisis  intervention, 
and  referral  services.  Some  problem  areas  required  more  specialized  services  than  the  EAP 
offered.  Clients  identifying  such  problem  areas  were  referred  to  other  community  resources  for 
further  services.  The  company  asked  the  EAP  to  monitor  referrals  in  order  to  examine  the  use 
of  community  resources.  If  trends  were  found  among  referrals,  the  company  would  consider 
inviting  specialists  to  conduct  noon-hour  workshops  for  all  staff. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

The  information  necessary  to  address  these  questions  was  routinely  documented  in  case  files. 
The  EAP  secretary  spent  20  hours  reviewing  case  files,  and  gathering,  organizing,  and  analyzing 
the  information. 


c.  How  were  the  data  collected? 

All  the  information  required  was  contained  in  the  clients’  files.  The  secretary  reviewed  each  case 
file  and  tabulated  the  information  required. 


d.  How  were  the  data  analyzed? 

The  information  was  tabulated  by  hand.  Harvard  Graphics  (see  Appendix  C)  was  used  to 
prepare  a bar  graph  to  illustrate  the  results  graphically. 
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Forty-eight  (36%)  of  the  133  employees  who  used  the  EAP  were  referred  to  community 
resources.  Table  1 indicates  the  number  of  people  referred  elsewhere,  grouped  by  the  primary 
problem  for  which  employees  attended  the  EAP.  Figure  1 shows  the  same  information  in  the 
form  of  a bar  graph  produced  using  Harvard  Graphics. 

As  shown  in  Table  1,  the  rate  of  use  of  other  services  was  highest  for  those  attending  the  EAP 
for  personal  substance  abuse  problems  (67%).  Overall,  only  about  25%  of  individuals 
addressing  issues  other  than  personal  substance  abuse  were  referred  to  services  in  the 
community. 


Figure  1 . Percent  of  Referrals  to  Community  Agencies  by  Primary 

Problem  Addressed 
by  Employees  Who  Attended  the  EAP 

% referred 

Presenting  problem  o%  20%  40%  60%  80%  100% 

Alcohol/drug  - self 
Alcohol/drug  - other 
Work  - related 
Personal  - family 
Personal  - financial 
Other 
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Table  1.  Numbers  and  Types  of  Referrals  to  Community  Agencies  by  Primary  Problem 

Addressed  by  Employees  Who  Attended  the  EAP 


Primary  problem 
addressed  at  EAP 

# and  % 
of  clients 
r^etred 

Agency  rrferred  to 
(and  # referred) 

Reason  for  referral 

alcohol/drug  use  - 
self  (n=36) 

24 

(67%) 

Treatment  Centre  A (5) 

Inpatient  treatment  for  substance 
abuse 

Community  Addictions  Treatment  B 
(19) 

Outpatient  treatment  for 
substance  abuse 

alcohol/drug  use  - 
other  (n=13) 

5 

(38%) 

Private  Therapist  (2) 

Family  Counselling  Centre  (3) 

Dealing  with  parental  substance 
abuse  throughout  childhood 

job  related 
(n=43) 

5 

(12%) 

Social  Services  Department,  General 
Hospital  (3) 

Stress  management  group 

Human  Rights  Commission  (2) 

Sexual  harassment  in  the 
workplace 

family  (n=20) 

8 

(40%) 

Family  Services  (2) 

Adolescent  and  Youth  Services  (3) 

Family  counselling 

Counselling  for  adolescent/ 
youth  dependent 

Women’s  Interval  Home  (2) 

Spousal  abuse 

YMCA  (1) 

Parenting  program 

financial 

(n=12) 

1 

(8%) 

Debt  Counselling  Centre  (1) 

Liaison  with  creditors 

other 

(n=9) 

5 

(56%) 

Private  psychiatrists  through  family 
physician  (2) 

Depression,  sleep  disorder 

Canadian  Mental  Health  Association 
(1) 

Group  therapy  - emotional 
health  issues 

Sexual  Assault  Centre  (2) 

Sexual  assault  outside  the 
workplace 

All  problems 
(n=133) 

48 

(36%) 
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The  results  demonstrated  that  the  majority  of  the  individuals  using  the  EAP  were  not  referred 
to  other  services.  Most  of  the  individuals  who  were  referred  to  community  resources  for  more 
intensive  services  were  addressing  personal  substance  abuse  problems.  This  finding  convinced 
the  company  to  set  up  an  in-house  substance  abuse  prevention  program.  They  have  decided  to 
ask  for  the  involvement  of  treatment  personnel  from  the  local  addictions  agencies  in  the  planning 
and  implementation  of  the  program. 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  heavy 


Part  3.  Client  Satisfaction 


To  what  extent  did  the  employees  find  the  EAP  helpful? 

To  what  extent  were  employees  satisfied  with  ike  EAP? 

To  what  extent  did  the  employees  feel  iha^  was  confidential? 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 

As  an  ongoing  quality  assurance  measure,  the  EAP  obtained  feedback  from  clients  regarding  the 
helpfulness  of  the  program  and  the  extent  to  which  clients  were  satisfied  with  the  services  they 
received.  As  well,  the  EAP  was  concerned  with  the  degree  to  which  the  clients  felt  that  the 
services  were  confidential. 


h.  What  resources  were  needed  to  collect  and  interpret  the  information? 

One  EAP  counsellor  was  designated  to  spend  about  100  hours  over  two  years  collecting  and 
organizing  the  information  required,  analyzing  the  data,  and  generating  a report. 


c.  How  were  the  data  collected? 

Each  employee  attending  the  EAP  was  asked  at  intake  for  permission  to  be  contacted  three 
months  after  termination  or  three  months  after  the  last  counselling  session  between  the  employee 
and  the  EAP.  The  follow-up  counsellor  asked  clients  a series  of  questions  about  their 
satisfaction  with  and  the  helpfulness  of  the  EAP  services.  The  questions  were  rated  on  a five- 
point  scale.  The  higher  the  score  (out  of  5),  the  more  positively  the  client  felt  about  the 
question.  As  well,  additional  comments  were  encouraged  and  recorded.  The  following  is  a 
sample  question: 
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Clients  were  asked  to  rate  the  helpfulness  of  four  aspects  of  the  program:  intake,  individual 
counselling,  family/marital  counselling,  and  referrals  to  other  programs.  They  were  also  asked 
to  rate  how  satisfied  they  were  with:  quality  of  service,  amount  of  help  received,  and  overall 
effectiveness  of  the  program.  Confidentiality  was  rated  on  a five-point  scale  from  1 - not  at  all 
confidential  (e.g.,  "others  found  out  information  about  me  from  the  program"),  to  5 - completely 
confidential  (e.g.,  "felt  secure  that  my  involvement  with  the  program  was  kept  completely 
private"). 


d.  How  were  the  data  analyzed? 

The  information  was  entered  in  the  computer  and  the  average  ratings  for  degree  of  satisfaction 
and  degree  of  helpfulness  were  computed.  Types  of  comments  were  also  tabulated. 

Bar  graphs  illustrating  the  results  were  prepared  using  Harvard  Graphics  (see  Appendix  C). 


e.  What  did  they  find  out? 

The  follow-up  counsellor  was  able  to  contact  128  of  the  133  individuals  who  had  used  the  EAP 
services  over  the  first  24  months  of  the  contract.  Table  1 summarizes  the  helpfulness  ratings 
of  the  different  types  of  services  provided  by  the  EAP.  Table  2 shows  the  results  of  the 
satisfaction  ratings  of  various  aspects  of  the  services  provided.  The  ratings  are  presented  using 
a different  format  (bar  graphs  using  Harvard  Graphics)  in  Figures  1 and  2. 

The  results  indicated  that  clients  found  the  EAP  to  be  mostly  helpful.  All  services  except  intake 
received  an  average  rating  of  4.0  or  higher,  indicating  a high  level  of  perceived  helpfulness. 

With  regards  to  client  satisfaction  with  the  EAP,  the  results  indicated  that  clients  were  generally 
satisfied  with  the  service.  Clients  were  generally  satisfied  with  the  quality  and  effectiveness  of 
the  service,  but  were  less  satisfied  with  the  quantity  of  the  services  available. 

Finally,  the  results  indicated  that  clients  perceived  the  EAP  to  be  very  confidential. 


EAP  Counselling  Program 


116 


Table  1.  Helpfulness  Ratings  of  the  Different  Types  of  Services  Offered  by  the  EAP 
and  Examples  of  Comments  Made 


T^pe  of  service 
recdved 

Average 

rating 

Examples  of  comments 

intake 

(n=128) 

3.6 

• didn’t  like  changing  counsellors  after  the  first  session 

• really  helpful  - was  all  I needed 

individual 

counselling 

(n=103) 

4.4 

• the  counsellor  helped  me  a lot 

• the  counsellor  made  me  feel  really  comfortable 

• the  counsellor  really  understood  me 

• the  counsellor  made  me  feel  like  an  idiot 

family/marital 

counselling 

(n=17) 

4.2 

• counsellor  shouldn’t  have  seen  my  husband  and  I together, 
as  I would  have  preferred  being  seen  separately 

• counsellor  let  my  kids  control  the  session 

referral  to  other 
programs 
(n=47) 

4.0 

• it  was  just  what  I was  looking  for 

• it  wasn’t  quite  what  I was  looking  for  and  I had  to  go 
somewhere  else 

• wish  I didn’t  have  to  go  see  someone  else  - not  enough 
sessions  with  the  EAP  counsellor 

Other  comments  made  were: 

• it  was  really  nice  that  someone  called  to  see  how  I was  doing  (follow-up) 

• I wish  I could  have  gone  back  once  or  twice  after  I stopped  going  on  a regular  basis,  just 
to  have  a little  extra  support 

• we  should  have  had  one  of  these  programs  years  ago 

• there  are  a lot  of  people  at  work  who  don’t  know  about  this 

• the  counsellor  I saw  told  me  more  about  his  problems  than  he  listened  to  my  problems. 
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Table  2.  Satisfaction  Ratings  of  Aspects  of  Services  and  Examples  of  Comments 


Aspect  of 
service  rated 

Average 

rating 

(n=128) 

Examples  of  comments 

quality  of  the 
service 

4.2 

• better  than  expected 

• this  guy  shouldn’t  be  doing  this  kind  of  work  - he  has  more 
problems  than  1 do 

amount  of  help 
received 

3.2 

• counselling  finished  just  when  I felt  comfortable  with  the 
counsellor 

• should  have  been  able  to  have  more  sessions 

overall 

effectiveness  of 
the  program 

4.2 

• the  program  was  really  helpful 

• I didn’t  know  a lot  of  the  information  the  counsellor  gave  me 

• I would  recommend  the  program  to  others 

• we  should  have  had  one  of  these  programs  years  ago 

confidentiality 

4.7 

• it  was  very  important  that  the  office  was  in  a different  part  of 
town  than  work 

• it  was  really  important  that  no  one  at  work  would  know  I was 
coming  here 

• it  was  important  that  not  even  my  family  would  know  why  I was 
coming  here 

• I hardly  told  the  counsellor  anything  because  I wasn’t  sure  that 
he’d  keep  it  to  himself 
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Figure  1 . Helpfulness  Ratings  of  the  Different  Types  of  Services 

Offered  by  the  EAP 


Rating  category  q 


Intake 


Individual  Counselling 
Family/Marital 

Referral 


Average  rating  (out  of  5) 
12  3 4 


Figure  2.  Satisfaction  Ratings  of  Different  Aspects  of  the  Services 

Offered  by  the  EAP 

Average  rating  (out  of  5) 

Rating  category  o 1 2 3 4 5 


Quality  of  service 


Amount  of  help 


Overall  effectiveness 


Confidentiality 
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/.  How  were  the  results  used? 

Many  of  the  additional  comments  made  were  positive  and  helpful.  The  EAP  made  some 
adjustments  in  their  procedures  as  a result  of  some  of  the  comments.  In  particular,  it  was 
decided  that  the  intake  worker  would  assess  clients  with  a brief  screening  interview  at  the  initial 
telephone  contact.  Clients  would  then  be  assigned  to  an  appropriate  counsellor  who  would 
complete  the  full  assessment  with  the  client  as  well  as  any  additional  sessions  needed.  This 
eliminated  the  need  to  deal  with  more  than  one  counsellor.  As  well,  a follow-up  component  was 
made  a standard  part  of  the  service. 

The  negative  comments  addressed  two  main  concerns:  the  limited  amount  of  service  available 
and  the  attitude  and  integrity  of  one  of  the  counsellors.  The  counsellor  who  had  been  perceived 
negatively  by  a number  of  clients  met  with  the  manager  of  the  EAP  regarding  job  performance. 
The  counsellor  had  only  recently  entered  the  counselling  field  and  had  come  to  the  conclusion 
on  his  own  that  he  was  not  well-suited  to  the  job.  The  feedback  from  clients  confirmed  his 
perceptions,  and  he  resigned  voluntarily  to  pursue  another  line  of  work.  To  address  the  issue 
of  the  amount  of  service  available,  the  company  agreed  to  extend  the  maximum  number  of 
sessions  from  four  to  six. 


Methodological,  measurement,  and  ethical  issues: 

Methodological.  In  this  example,  follow-up  contact  was  done  by  telephone  since  this  would 
probably  elicit  a higher  rate  of  response  than  would  have  been  achieved  using  mailed-out 
questionnaires.  Another  benefit  of  a personal  interview,  face  to  face  or  by  telephone,  is  that 
additional  comments  and  suggestions  can  be  encouraged. 

There  were  also  drawbacks  to  telephone  interviewing.  People  are  somewhat  more  likely  to  rate 
services  positively  when  interviewed  personally,  as  this  type  of  information  is  not  completely 
anonymous.  A mailed  questionnaire  can  be  completely  anonymous,  and  so  people  are  more 
likely  to  indicate  their  true  thoughts  and  feelings. 

For  the  purposes  of  this  evaluation,  the  EAP  chose  to  contact  people  by  telephone  as  they  felt 
that  the  higher  rate  of  response  and  the  increased  amount  of  feedback  would  outweigh  the  risk  of 
positive  bias.  In  addition,  the  follow-up  worker  was  instructed  to  emphasize  with  clients  that  the 
program  was  interested  in  their  honest  responses  and  that  names  would  not  appear  with  any 
questionnaire. 

Measurement.  The  use  of  a five-point  scale,  rather  than  yes/no  ratings,  allowed  clients  to  more 
accurately  indicate  the  degree  to  which  they  found  the  program  helpful,  satisfactory,  and 
confidential. 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  moderate 


Part  4.  Program  Delivery  by  Client  Satisfaction 


What  proportion  of  clients  followed  through  on  referrals  to  different 
community  agencies? 

How  satisfied  were  clients  with  services  received  from  community  agencies 
to  which  they  had  been  referred  by  the  EAP  counsellor? 

How  helpful  did  clients  find  the  services  to  which  they  had  been  referred? 

To  what  extent  did  the  community  agencies  report  that  the  referrals  to 
them  had  been  appropriate? 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 

An  earlier  evaluation  had  examined  the  community  resources  to  which  referrals  were  made  to 
supplement  EAP  services  (see  Part  2).  The  EAP  provider  wanted  to  determine  the  proportion 
of  clients  who  used  the  services  of  the  community  agencies  to  which  they  had  been  referred. 
As  well,  the  EAP  provider  was  interested  in  the  clients’  level  of  satisfaction  with  those  services. 
Finally,  the  EAP  provider  felt  that  it  was  also  important  to  assess  the  agencies’  perceptions  of 
the  appropriateness  of  the  referrals  made  to  them  by  the  EAP.  Thus,  in  the  next  year  of 
operations,  the  EAP  provider  instituted  a plan  for  monitoring  referrals. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

The  information  necessary  to  address  these  questions  was  collected  routinely  during  the  follow- 
up telephone  interview  with  the  client.  In  addition,  as  part  of  the  referral  procedure,  a follow-up 
telephone  call  was  made  to  the  community  resource  to  which  a referral  had  been  made.  A staff 
person  of  the  EAP  spent  eight  hours  reviewing  case  files,  organizing  and  analyzing  the 
information,  and  writing  a report. 
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c.  How  were  the  data  collected? 

Clients  were  routinely  contacted  for  a follow-up  telephone  interview  with  the  EAP  follow-up 
worker  three  months  after  the  contact  between  the  client  and  the  EAP.  Prior  to  conducting  a 
follow-up  interview,  the  follow-up  worker  reviewed  the  client’s  case  file  in  order  to  determine 
whether  a referral  had  been  made  to  an  outside  resource.  Clients  for  whom  a referral  had  been 
made  were  asked  the  following:  whether  they  had  attended  at  least  one  appointment  at  the 
agency  to  which  they  had  been  referred,  whether  they  found  services  from  that  agency  generally 
helpful,  whether  they  were  generally  satisfied  with  the  service  they  received,  and  the  extent  to 
which  they  felt  that  the  referral  was  appropriate  for  them.  The  latter  three  questions  were  asked 
on  a five-point  scale. 

When  a referral  was  made  to  an  outside  resource,  the  EAP  requested  that  clients  sign  a release 
of  information  form  that  would  allow  the  EAP  to  exchange  information  about  the  client  with  the 
agency  to  which  he  or  she  had  been  referred.  At  the  time  of  the  referral,  the  EAP  counsellor 
made  note  of  the  scheduled  appointment.  Within  one  week  following  the  scheduled  appointment, 
the  EAP  counsellor  contacted  the  agency  to  which  the  referral  had  been  made  in  order  to  find 
out  if  the  client  attended,  and  whether  the  agency  saw  the  referral  as  appropriate. 


d.  How  were  the  data  analyzed? 

Responses  from  the  client  follow-up  and  the  agency  follow-up  were  tabulated  by  hand. 


e.  What  did  they  find  out? 

Thirty-one  clients  were  referred  to  other  agencies  for  further  services.  Of  these,  two  people 
refused  to  give  consent  to  any  release  of  information  by  the  EAP  to  the  agency  or  vice  versa, 
and  one  person  consented  only  to  the  EAP  releasing  information  to  the  agency  about  the  client, 
but  did  not  consent  to  a reciprocal  release  of  information.  Therefore,  agency  follow-up  contacts 
were  made  for  only  28  of  the  31  referrals. 

Follow-up  interviews  were  completed  with  30  of  the  31  clients  who  had  been  referred  elsewhere. 
The  one  person  who  could  not  be  located  at  follow-up  had  also  refused  to  sign  a release  of 
information;  therefore,  no  information  was  available  regarding  whether  this  person  followed 
through  on  the  referral.  Of  the  30  clients  who  had  been  referred  elsewhere  and  who  participated 
in  the  follow-up  interview,  26  (84%)  reported  that  they  had  attended  at  least  one  appointment 
with  the  agency  to  which  they  had  been  referred.  Follow-up  data  from  the  agencies  indicated, 
however,  that  there  was  no  agency  record  confirming  attendance  at  the  program  by  two  of  the 
people  who  said  they  had  attended  at  least  one  appointment.  Therefore,  the  ratings  of  the 
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satisfaction,  and  appropriateness  of  referrals  are  shown  in  Table  1,  based  on  the  24  cases 
included  in  the  analyses. 


Table  1. 


Ratings  Made  by  Clients  Who  Attended  at  Least  One  Appointment  at  the 
Agency  to  Which  They  Had  Been  Referred 


Average 

Rating 

helpfulness  of  services  received  from  outside  agency 

4.2 

satisfaction  with  services  received  from  outside  agency 

4.0 

how  appropriate  the  client  felt  the  referral  was 

4.3 

For  the  four  individuals  who  reported  that  they  did  not  attend  at  least  one  appointment,  the 
following  reasons  were  reported:  one  individual  was  referred  to  outpatient  addictions  treatment 
but  decided  to  attend  AA  and  try  to  quit  drinking/drug  using  without  formal  outpatient  treatment; 
one  individual  was  referred  to  an  inpatient  addictions  treatment  program  but  did  not  abstain  from 
drug  use  for  the  required  length  of  time  prior  to  the  scheduled  appointment,  and  so  did  not 
attend;  one  individual  referred  to  the  Women’s  Interval  Home  decided  to  return  to  her  husband; 
one  individual  was  referred  to  Adolescent  and  Youth  Services  for  counselling  for  dependent 
children,  but  the  child  refused  to  attend. 

Of  the  28  clients  for  whom  follow-up  contact  was  made  with  the  agency  to  which  they  had  been 
referred,  24  were  confirmed  as  attending  at  least  one  appointment.  Responses  from  the  agency 
indicated  that  21  (88%)  of  these  referrals  were  considered  appropriate. 


/.  How  were  the  results  used? 

Most  clients  who  were  referred  elsewhere  for  further  services  reported  that  they  attended  at  least 
one  appointment  at  the  referral  destination.  Clients  were  generally  satisfied  with  the  services 
they  received  from  outside  referrals  and  they  found  the  services  helpful. 

The  agency  follow-up  records  confirmed  that  most  clients  followed  through  on  referrals  and  that 
most  referrals  were  appropriate.  The  four  referrals  that  were  considered  inappropriate  were 
discussed  at  a staff  meeting  so  that  similar  mistakes  would  not  be  made  in  the  future. 
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The  numbers  were  too  small  to  calculate  satisfaction  ratings  with  the  different  types  of  agencies 
to  which  clients  were  referred.  The  EAP  provider  decided  to  continue  to  monitor  satisfaction 
with  referrals  as  part  of  regular  follow-up  in  order  to  identify  those  agencies  for  which  clients 
gave  the  highest  ratings. 


Methodological,  measurement,  and  ethical  issues 

Ethical.  Inter-communication  between  agencies  regarding  specific  clients  required  that  a 
release  of  information  be  granted  by  the  client. 
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G.  A Community  Support  Program  for  Immigrant  Women 


Program:  This  is  a two-year  pilot  program  designed  to  provide  practical  and  emotional 
support  to  women  in  the  community  who  have  arrived  in  Canada  during  the  past  five  years.  The 
program  is  targeted  primarily  at  women  who  are  socially  (or  physically)  isolated  from  the 
broader  community  and  who  may  be  at  risk  of  substance  abuse,  particularly  prescription  drug 
abuse.  The  program  is  located  in  a city  where  knowledge  of  English  is  necessary  to  function 
independently. 

The  program  components  include:  providing  information  and  assistance  to  clients  who  might 
wish  to  enrol  in  English  as  a Second  Language  (ESL)  training,  assistance  in  locating  suitable 
employment,  a support  group  where  participants  discuss  problems  relating  to  their  experience 
as  immigrants  (including  problems  relating  to  prescription  drug  use),  assistance  in  linking 
participants  to  ethnic  clubs/organizations  in  the  community,  and  an  awareness  and  information 
program  designed  to  improve  access  to  and  use  of  community  helping  agencies  (including  those 
focusing  on  substance  use). 

The  impetus  for  the  program  comes  from  research  findings  indicating  that  moving  to  a new 
country  often  leads  to  increased  social  isolation  which  in  turn  can  lead  to  elevated  levels  of  stress 
and  substance  use.  Immigrant  women  are  particularly  vulnerable  to  abuse  of  prescription  drugs 
such  as  sedatives  and  tranquilizers.  On  average,  they  have  lower  levels  of  education  than  male 
immigrants,  are  less  likely  to  have  paid  employment,  are  less  likely  to  have  close  friends  or 
relatives  in  the  host  country,  and  are  less  likely  to  speak  English. 

This  example  includes  quality  enhancement  questions  for  three  levels  of  evaluation: 


Part  1. 

Logic  Model 

Part  2. 

Client  Characteristics 

Part  3. 

Outcome  Evaluation. 
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Part  1.  Program  Logic  Model 

Program  Logic  Model  for  a Community 
Support  Program  for  Immigrant  Women 
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Level  of  difficulty:  easy 

Level  of  resources  needed:  moderate 


Part  2.  Client  Characteristics 


To  what  extent  (Ud  r^gM  women  differ  fwm  non-r^gee  women  in 
terms  (rfsociM  isolddony  mg  and  tranquilizers,  and  other 

factors  that  might  affect  their  adjustment? 


a.  Who  was  asking  the  question  and  why  did  they  want  this  information? 

The  program  had  been  designed  to  address  the  needs  of  immigrant  women  generally,  with  no 
particular  emphasis  on  immigrant  refugee  women.  However,  over  the  course  of  the  two-year 
period  prior  to  the  program  beginning,  the  community  had  absorbed  a sizable  influx  of  refugees. 
Thus,  a fairly  large  proportion  of  program  participants  were  refugees.  Program  staff  had 
observed  that  refugee  women  who  attended  the  program  seemed  to  have  greater  problems 
adjusting  than  women  who  immigrated  to  this  country  for  other  reasons.  In  particular,  many 
of  the  refugee  women  had  suffered  political  persecution  in  their  homeland,  had  been  deprived 
of  their  home  and  personal  possessions,  had  lost  contact  with  family  and  friends,  had  witnessed 
the  murder  of  a family  member,  and  often  had  been  severely  beaten  and  tortured. 

Because  of  these  factors  and  the  involuntary  nature  of  their  move,  program  personnel  wondered 
whether  refugee  women  compared  to  other  immigrant  women  were  significantly  less  educated, 
had  less  knowledge  of  English,  and  had  fewer  contacts  with  individuals  and  organizations  in  the 
community.  Similarly,  it  appeared  likely  that  refugee  women  would  be  more  at  risk  for  abuse 
of  sleeping  pills  and  tranquilizers  because  of  past  trauma  as  well  as  current  isolation. 

If  refugee  women  were  found  to  be  considerably  more  socially  isolated  and  more  likely  to  use 
psychoactive  prescription  drugs  than  non-refugee  women,  the  program  might  need  to  be  adjusted 
to  meet  their  special  needs.  In  particular,  refugee  women  might  need  additional  counselling  and 
other  services.  In  planning  the  program  for  the  next  year,  consideration  needed  to  be  given  to 
the  proportion  of  refugees  in  the  target  population  in  order  to  estimate  the  program’s  caseload 
capacity.  Other  adjustments  to  the  program  might  also  need  to  be  made,  especially  in  terms  of 
helping  refugee  women  deal  with  past  trauma. 
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b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

The  outreach  workers  collected  the  information  as  part  of  intake.  It  took  one  staff  member  two 
days  to  enter  the  data  on  the  computer,  analyze  the  data  and  prepare  a report. 


c.  How  were  the  data  collected? 

Outreach  workers  completed  a short  client  intake  form  for  each  woman  who  became  involved 
with  the  program.  The  intake  form  included  the  following  information  relevant  to  the  present 
question:  average  length  of  time  in  country;  whether  the  person  lives  alone;  age;  marital  status 
(including  whether  living  with  spouse  or  partner);  education  level;  number  of  contacts  with 
friends  or  relatives  (aside  from  spouse  and  children)  in  past  month;  and  whether  the  person: 
speaks  English,  had  paid  employment  in  this  country  during  the  past  12  months,  is  affiliated  with 
any  ethnic  club/organization,  is  aware  of  helping  agencies,  used  helping  agencies  in  last  12 
months,  used  sleeping  pills  or  tranquilizers  during  the  past  month,  or  used  tranquilizers  or 
sleeping  pills  daily  or  almost  daily  during  the  past  month.  The  form  also  included  the  person’s 
usual  frequency  of  drinking  during  the  past  month,  the  number  of  times  the  person  had 
consumed  five  or  more  drinks  on  any  occasion  during  the  past  six  months,  and  whether  the 
person  used  illicit  drugs  during  the  past  six  months.  (A  sample  intake  form  is  provided  at  the 
end  of  this  part  of  the  Example.) 


d.  How  were  the  data  analyzed? 

The  data  were  entered  on  the  computer  and  the  characteristics  of  program  participants  were 
calculated  for  all  program  participants,  as  well  as  for  refugees  and  non-refugees  separately. 
Frequencies  and  percentages  were  calculated  for  categorical  variables  and  means  for  quantitative 
variables.  The  median  (i.e.,  mid-point)  was  calculated  for  education  level. 

Characteristics  of  refugee  and  non-refugee  participants  are  shown  in  graphs  prepared  using 
Harvard  Graphics  (see  Appendix  C). 


e.  What  did  they  find  out? 

As  suspected  by  the  program  staff,  refugee  participants  were  more  likely  to  be  socially  isolated 
and  have  less  resources  available  to  them  (not  living  with  spouse  or  partner,  living  alone,  lower 
education,  in  the  country  slightly  less  time,  less  likely  to  have  functional  English,  less  likely  to 
have  had  paid  employment,  less  likely  to  belong  to  ethnic  organization,  and  less  likely  to  use 
helping  agencies).  (See  Table  1 for  detailed  results  and  Figures  1 and  2 for  a pie  chart  and  bar 
graph  showing  graphic  display  of  findings,  prepared  using  Harvard  Graphics.) 
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Table  1.  Characteristics  of  Refugee  and  Non-refugee  Participants  in  the  Community 
Support  Program  for  Immigrant  Women 


Refugee 

n=60 

Non-refugee 

n=140 

All 

participants 

n=200 

average  age 

32  years 

37  years 

36  years 

married  (lives  with  spouse) 

30% 

60% 

51% 

married  (does  not  live  with  spouse) 

30% 

10% 

16% 

not  married  (single,  separated,  divorced,  widowed) 

40% 

30% 

33% 

lives  alone 

40% 

25% 

30% 

median  education  level 

grade  7 

grade  10 

grade  9 

average  number  of  years  in  country 

2.2  years 

2,6  years 

2.5  years 

speaks  English  well  enough  to  shop,  work  and 
socialize  in  English-speaking  environment 

30% 

60% 

51% 

has  had  paid  employment  in  this  country  during 
past  12  months 

17% 

50% 

41% 

belongs  to  an  ethnic  club  or  organization 

20% 

43% 

36% 

has  used  helping  agencies  during  past  12  months 

3% 

15% 

12% 

Figure  1 . Marital  Status  of  Refugee  and  Non-refugee  Participants  in 
the  Community  Support  Program  for  Immigrant  Women 


Marr1ed-w/spouse 


Marriod-not  w/spouae 
30% 

Refugee  (n=:60) 


Married-w/spouse 


60% 


Not  married 
40% 


Married-not  w/apouae 
10% 

Non-refugee  (n=140) 
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Figure  2.  Characteristics  of  Refugee  and  Non-refugee  Participants 
in  the  Community  Support  Program  for  Immigrant  Women 

Characteristics 
Lives  alone 

Functional  English 
Paid  employment 
Ethnic  club 
Aware  of  help 
Used  help 

■ Refugee  (n«60)  ^Non-refugee  (n«140) 


Percentage  of  women 


With  regards  to  use  of  drugs  and  alcohol,  refugee  participants  were  more  likely  to  use  sleeping 
pills  and  tranquilizers  than  non-refugee  participants,  with  50%  of  refugee  women  who  came  to 
the  program  using  tranquilizers  or  sleeping  pills  daily  or  almost  daily.  Alcohol  and  illicit  drug 
use  was  low  and  did  not  differ  greatly  between  the  two  groups  (see  Table  2 and  Figure  3). 
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Table  2.  Drug  and  Alcohol  Use  by  Refugee  and  Non-refugee  Participants  in  the 
Community  Support  Program  for  Immigrant  Women 


Refugee 

Non-refugee 

n=140 

All 

participants 

n=200 

used  tranquilizers  and/or  sleeping  pills  during  past 
six  months 

60% 

35% 

43% 

used  tranquilizers  and/or  sleeping  pills  daily  or 
almost  daily  during  past  month 

50% 

14% 

25% 

consumed  any  alcohol  during  past  six  months 

60% 

55% 

57% 

used  both  alcohol  and  tranquilizers  and/or  sleeping 
pills  during  past  six  months 

43% 

21% 

28% 

consumed  alcohol  daily  or  almost  daily  during  past 
six  months 

5% 

3% 

4% 

consumed  more  than  five  drinks  on  one  occasion  at 
least  six  times  during  past  six  months 

3% 

3% 

3% 

used  illicit  drugs  during  past  six  months 

2% 

0% 

<1% 

Figure  3.  Drug  and  Alcohol  Use  by  Refugee  and  Non-refugee 
Participants  in  the  Community  Support  Program  for  Immigrant 

Women 


% of  women  reporting  described  use 

Descriptions  Of  drug  use  o%  10%  20%  30%  40%  50%  60%  70% 


Tranquilizers 
Tranquilizers 
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/.  How  were  the  results  used? 

The  results  strongly  supported  the  conclusion  that  refugee  women  were  more  socially  isolated 
than  non-refugee  immigrant  women  on  all  of  the  indicators  monitored  in  the  present  evaluation. 
It  seemed  plausible  that  isolation  was  a contributing  factor  to  the  large  proportion  of  women  in 
this  group  who  used  prescription  drugs  on  a daily  basis.  The  greater  needs  of  refugee 
participants  will  be  considered  in  decisions  made  about  caseloads,  programming  and  budget  for 
the  coming  year. 


Methodological,  measurement  and  ethical  issues: 

Methodological.  Program  staff  became  aware  that  the  collection  and  entry  of  data 
pertaining  to  the  various  indicators  of  social  isolation  at  the  time  of  client  intake  could  be 
used  as  pre-test  or  baseline  data  in  an  outcome  evaluation  of  the  program  in  the  future. 
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File  # 

Community  Support  Program  for  Immigrant  Women 
Client  Intake 


Date  of  assessment:  

Name:  

Address:  

Telephone:  (Home):  (Work): 

Native  language  (mother  tongue):  

Other  languages  spoken:  

1 . Date  of  birth:  

2.  Marital  status: 

I I Married  or  common-law  Q 

I I Separated  Q 

I I Divorced 

3.  Current  living  arrangements: 

□ Living  alone 

I I Living  with  spouse/partner  Q 

I I Living  with  other  family  members 

4.  Highest  level  of  education  attained: 

(Try  to  estimate  equivalent  grade  or  university  degree  in  Canadian  education  system.) 

5.  Month  and  year  of  arrival  of  Canada:  

Number  of  years  in  Canada:  years 

(Express  months  as  a decimal  fraction  of  a year.  For  example  if  the  person  had  been  in  Canada  two  years 
and  six  months,  this  would  be  noted  as  2.5  years.) 

6.  Status  upon  arrival  in  Canada: 

□ Landed  immigrant 

I I Refugee 

I I Student 


Widowed 

Single 


□ Living  with  friends 

Other  {specify) 
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7 . Functional  English:  ( Can  understand  and  speak  English  well  enough  to  respond  to  questionnaires  on  intake 
form  without  help  of  a translator.) 

□ Yes 

□ No 

8.  Paid  employment  in  Canada  during  past  12  months: 

□ Yes,  (specify) 

□ No 

9.  Affiliation  with  an  ethnic  group/club/organization: 

□ Yes,  (specify) 

□ No 

10.  Involvement  with  community  agencies  during  the  last  12  months: 

(Please  check  all  that  apply.) 

I I Family  Services 

□ Children’s  Aid  Society 

I I Sexual  Assault  Crisis  Centre 

I I Women’s  Community  House 

□ Community  Mental  Heath 

I I Public  Health  Unit 

I I Other  (specify) 

11.  Alcohol  and  other  drug  use  during  the  past  six  months: 


□ Yes 

□ No 

any  use  of  tranquilizers  and/or  sleeping  pills 

□ Yes 

□ No 

tranquilizers  and/or  sleeping  pills  used  daily  or  almost  daily 

□ Yes 

□ No 

any  use  of  alcohol 

□ Yes 

□ No 

alcohol  consumed  daily  or  almost  daily 

□ Yes 

□ No 

has  had  more  than  five  drinks  on  one  occasion  at  least  six  times  during  past  six 
months 

□ Yes 

□ No 

any  use  of  illicit  pills 
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Level  of  difficulty:  easy 

Level  of  resources  needed:  moderate 


Part  3.  Outcome  Evaluation 


What  proportion  of  clients  were  employed  12  months  after  starts 
program? 

What  proportion  of  clients  were  affiliated  with  ethnic  clubs/ organiz/ations 
12  months  after  starting  the  pmgram  ? 

What  proportion  of  cUents  used  other  community  services  during  the  12 
months  after  starting  the  program  ? 

What  proportion  of  clients  were  using  sleeping  pills  and/or  tranquilizers 
12  months  after  starting  the  program? 


fl.  Who  was  asking  the  question  and  why  did  they  want  this  information? 

In  order  to  argue  for  continued  funding  the  program  needed  to  show  some  evidence  of  success. 
While  they  did  not  have  the  resources  to  conduct  a full  scale  evaluation,  it  was  possible  to  assess 
the  extent  to  which  some  of  the  outcome  objectives  were  being  achieved.  In  particular,  they 
could  compare  pre-  and  post-  functioning  of  program  participants  in  terms  of  employment, 
affiliation  with  ethnic  organizations,  use  of  community  services,  and  use  of  sleeping  pills  and/or 
tranquilizers. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

The  program  secretary  spent  six  hours  retrieving  the  follow-up  summaries  from  case  files, 
entering  the  information  into  the  computer,  and  preparing  frequencies. 
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c.  How  were  the  data  collected? 

The  program  includes  many  volunteers  from  the  various  ethnic  communities.  As  part  of 
program  operations,  volunteers  telephoned  program  participants  12  months  after  they  first  came 
to  the  program  to  find  out  how  they  were  doing  and  if  they  needed  any  further  assistance.  At 
this  point,  some  basic  statistics  were  collected  regarding  whether  or  not  they  were  employed, 
whether  they  belonged  to  an  ethnic  organization,  whether  they  had  used  any  community  services, 
and  whether  they  had  taken  sleeping  pills  and/or  tranquilizers.  The  information  was  recorded 
on  a follow-up  summary  sheet. 


d.  How  were  the  data  analyzed? 

The  results  were  entered  into  the  computer  and  analyzed  using  a statistical  software  program. 
Frequencies  of  responses  were  computed  for  answers  regarding  employment,  ethnic  organization, 
use  of  community  services,  and  use  of  sleeping  pills  and/or  tranquilizers. 

Selected  characteristics  of  participants  are  shown  in  line  graph  form  comparing  status  at  initial 
assessment  and  at  12-month  follow-up  (prepared  using  Harvard  Graphics  [see  Appendix  C]). 


e.  What  did  they  find  out? 

Of  2(X)  women  assessed,  184  (92%)  participated  in  the  follow-up  interview.  Of  the  16  who  did 
not  participate,  15  could  not  be  located  and  one  did  not  wish  to  be  interviewed.  As  shown  in 
Table  1,  the  program  participants  improved  on  all  indicators,  with  substantial  increases  in 
employment,  affiliation  with  an  ethnic  club  and  use  of  available  services.  In  addition,  use  of 
tranquilizers  and/or  sleeping  pills  dropped  dramatically.  The  results  are  shown  using  a line 
graph  in  Figure  1 (produced  using  Harvard  Graphics). 
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Table  1.  Percent  of  Participants  in  the  Community  Support  Program  for  Immigrant 
Women  Who  were  Employed,  Who  Belonged  to  an  Ethnic  Organization,  Who 
Used  Community  Services,  and  Who  Used  Tranquilizers  and/or  Sleeping  Pills 
(Initial  Status  Compared  to  Status  at  12-Month  Follow-Up) 


Initial 

status 

n=200 

Status  at 

12  months 
n=184 

has  had  paid  employment  during  past  12  months 

41% 

65% 

currently  belongs  to  an  ethnic  club  or  organization 

36% 

70% 

has  used  helping  agency  during  past  12  months 

12% 

23% 

used  tranquilizers  and/or  sleeping  pills  during  past  six  months 

43% 

19% 

Figure  1 . Change  in  Status  Regarding  Employment,  Belonging  to  an  Ethnic 
Club,  Using  Community  Services,  and  Using  Tranquilizers  and/or 
Sleeping  Pills  (Initial  Status  Compared  to  12-Month  Follow-up) 


Characteristics  examined 
Have  paid  employment  ■'  Belong  to  ethnic  club 
Used  helping  agencies  — Used  tranquilizers 
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/.  How  were  the  results  used? 

The  results  indicated  substantial  change  on  all  three  measures.  These  results  were  used  to 
demonstrate  that  the  program  was  succeeding  in  achieving  at  least  some  of  the  short-term 
objectives  as  well  as  showing  success  regarding  the  long-term  objective  of  reducing  substance 
abuse. 


Methodological,  measurement  and  ethical  issues: 

Methodological.  The  nature  of  the  program  was  such  that  participants  spoke  many 
languages  and  many  participants  did  not  have  a good  command  of  English.  Therefore,  the 
multilingual  volunteers  were  essential  to  the  functioning  of  the  program  as  well  as  to  the 
follow-up.  Because  the  person’s  native  language  was  recorded  on  the  intake  form,  it  was 
possible  for  program  staff  to  identify  the  language  potentially  needed  by  the  volunteer  in 
order  to  conduct  the  follow-up.  There  were  many  languages  represented  among 
volunteers  and,  of  course,  many  program  participants  were  able  to  be  interviewed  in 
English.  Ensuring  that  respondents  were  able  to  understand  and  respond  to  the  follow-up 
questions  was  a major  concern  of  the  program. 

Because  there  was  no  control  or  comparison  group,  it  is  not  possible  to  know  for  sure  that 
changes  made  by  the  clients  can  be  attributed  to  the  program.  While  these  results  are 
encouraging  and  suggest  that  the  program  has  been  helpful,  it  is  possible  that  the 
participants  would  have  shown  improvement  in  these  areas  without  attending  the  program. 
To  conclusively  assess  the  overall  effectiveness  of  the  program,  it  would  be  necessary  to 
include  a control  group  in  the  evaluation  (i.e.,  a group  of  people  who  were  exactly  like 
the  participants  of  the  program  except  that  they  did  not  receive  the  program). 
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Program:  The  program  is  one  component  of  Youth  Addiction  Services.  Clients  range  in  age 
from  16  to  24  and  are  referred  to  the  service  by  a variety  of  sources  such  as  the  legal  system, 
their  family /friends,  their  school,  and  themselves.  Some  clients  are  also  referred  by  other 
sources,  such  as  employers,  physicians,  and  social  services  agencies. 

The  four  session  prevention  program  is  targeted  at  youth  who  experience  a low  level  of 
dependency  on  alcohol  and/or  other  drugs  and  a low  level  of  problems  associated  with  the  use 
of  alcohol  and/or  other  drugs.  These  youths  have  been  referred  to  Youth  Addiction  Services  as 
a result  of  alcohol  and/or  other  drug  use  behaviour  about  which  they  or  others  are  concerned. 
The  prevention  program  consists  of  four  educational  sessions.  Each  session  involves  a 
combination  of  videos,  discussion,  and  handouts.  The  sessions  are:  1.  Alcohol  and  its  effects; 
2.  Other  drugs  and  their  effects;  3.  Why  people  use  alcohol  and  other  drugs;  4.  Making 
healthier  choices  than  to  use  alcohol  and  other  drugs. 

This  example  includes  quality  enhancement  questions  at  three  levels  of  evaluation: 

Part  1.  Client  Characteristics 


Part  2.  Client  Satisfaction 


Part  3.  Client  Characteristics  by  Client  Satisfaction. 
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Level  of  difficulty:  easy 

Level  of  resources  needed:  moderate 


Part  1.  Client  Characteristics 

Has  the  number  of  referrals  from  schools  and  probation  setyice:s  increased 
from  the  previous  year? 

What  proportion  of  clients  are  aged  16-19? 

What  is  the  average  age  of  program  participaMs  from  each  referral 
source? 


a.  Who  was  asking  the  question  and  why  did  they  want  this  information? 

The  Program’s  Community  Advisory  Committee  requires  the  program  manager  to  monitor 
the  characteristics  of  those  attending  the  program  in  order  to  ensure  that  the  program  content  and 
delivery  is  suited  to  the  participants  and  that  other  agencies  in  the  community  are  aware  of  and 
using  the  service. 

In  the  last  year,  the  program  has  been  working  together  with  schools  and  with  probation  services 
to  identify  high-risk  youth.  Therefore,  they  expect  increased  referrals  from  these  sources. 

In  previous  years,  the  majority  of  program  participants  have  been  in  the  16-19  age  range. 
Therefore,  most  of  the  program  materials  used  (e.g.,  films/videos,  handouts)  have  been 
specifically  developed  for  this  age  group.  It  is  important,  however,  for  the  program  to  monitor 
the  age  range  of  participants  to  ensure  that  the  program  materials  continue  to  be  appropriate. 
While  the  school  referrals  are  likely  to  be  in  this  age  group,  it  is  possible  that  referrals  from 
other  agencies  may  be  older.  Therefore,  the  Advisory  Committee  wanted  to  monitor  not  only 
overall  average  age,  but  also  average  age  from  each  referral  source. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

Information  is  gathered  on  an  ongoing  basis.  The  program  secretary  spent  approximately  three 
hours  entering  information  into  the  computer,  tabulating  statistics  using  EPI  INFO  (see  Appendix 
A),  and  preparing  summary  tables. 
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c.  How  were  the  data  collected? 

As  each  client  is  admitted  to  the  prevention  program,  counsellors  record  his  or  her  gender,  age, 
and  referral  source  on  a program  participation  log.  At  the  end  of  the  year,  all  logs  are 
forwarded  to  the  program  secretary  for  tabulation. 


d.  How  were  the  data  analyzed? 

The  information  was  entered  and  analyzed  using  EPI  INFO  (see  Figure  1 for  command 
statements  in  EPI  INFO  for  doing  these  analyses).  Number  and  percent  of  clients  from  each 
type  of  referral  source  was  calculated  and  compared  (in  a table)  to  last  year’s  statistics.  Average 
age  overall  was  calculated,  as  well  as  average  age  from  each  referral  source. 

A line  graph  comparing  the  current  year’s  statistics  with  last  year’s  statistics  was  generated  using 
Lotus  1-2-3  (see  Appendix  B). 


e.  What  did  they  find  out? 

Forty-six  clients  attended  the  prevention  program  in  1992.  Table  1 shows  the  number  and 
percentage  of  referrals  from  each  referral  source  in  1992  compared  to  1991.  These  same  results 
are  shown  in  bar  graph  form  in  Figure  2,  produced  using  Lotus  1-2-3  (see  Appendix  B).  These 
results  indicate  that  the  number  of  referrcils  from  the  legal  system  and  from  schools  had 
increased  substantially  from  the  previous  year. 

The  age  of  participants  ranged  from  16  to  23  with  87%  of  participants  in  the  16-19  age  group. 
The  average  age  overall  was  17.8  years.  Table  2 shows  the  average  age  of  participants  from 
each  referral  source.  As  is  evident  from  this  table,  average  age  did  not  vary  greatly  according 
to  referral  source,  although  school  referrals  tended  to  be  slightly  younger,  on  average,  and  legal 
and  "other"  referrals  to  be  somewhat  older. 
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Figure  1.  Commands  and  Outputs  From  EPI  INFO  Used  to  Compute  Frequencies  of 
Referral  Sources  and  Mean  Age  and  Age  by  Referral  Source 


By  using  the  command,  FREQ  REFERRAL,  the  following  frequency  table  is  generated: 


REFERRAL 

j Freq 

Percent 

Cum. 

1 

1 8 

17.4% 

17.4% 

2 

1 12 

26.1% 

43.5% 

3 

1 14 

30.4% 

73.9% 

4 

1 7 

15.2% 

89.1% 

5 

i 5 

10.9% 

100.0% 

Total 

1 46 

100.0% 

Sum 

= 127.00 

Mean 

= 

2.76 

Standard 

deviation 

1 = 

1.23 

By  using  the  command,  MEANS  AGE,  the  following  frequency  table  and  statistics  are  generated: 


AGE  i 

Freq 

Percent 

Cum. 

16  1 

21 

45.7% 

45.7% 

17  1 

7 

15.2% 

60.9% 

18  1 

5 

10.9% 

71.7% 

19  j 

7 

15.2% 

87.0% 

20  1 

1 

2.2% 

89.1% 

21  1 

4 

8.7% 

97.8% 

23  1 

1 

2.2% 

100.0% 

Total  1 

46 

100.0% 

Sum 

= 

805.00 

Mean 

= 

17.50 

Standard 

deviation  = 

1.83 

By  using  the  command,  MEANS  AGE  REFERRAL  IN,  the  following  statistics  are  generated: 

MEANS  of  AGE  for  each  category  of  REFERRAL 


REFERRAL 

Obs 

Total 

Mean 

Variance 

Std  Dev 

1 

8 

139 

17.375 

3.982 

1.996 

2 

12 

216 

18.000 

3.091 

1.758 

3 

14 

234 

16.714 

0.835 

0.914 

4 

7 

124 

17.714 

6.905 

2.628 

5 

5 

92 

18.400 

5.300 

2.302 

REFERRAL 

Minimum 

25%ile 

Median 

75%ile 

Maximum 

Mode 

1 

16.000 

16.000 

16.000 

19.000 

21.000 

16.000 

2 

16.000 

16.500 

18.000 

19.000 

21.000 

16.000 

3 

16.000 

16.000 

16.500 

17.000 

19.000 

16.000 

4 

16.000 

16.000 

16.000 

19.000 

23.000 

16.000 

5 

16.000 

16.000 

19.000 

20.000 

21.000 

16.000 

ANOVA 

(For  normally  distributed  data  only) 


Variation 

SS 

df 

MS 

F statistic 

p-value 

Between 

16.139 

4 

4.035 

1.222 

0.315909 

Within 

135.361 

41 

3.301 

Total 

151.500 

45 

Bartlett's 

test  for  homogeneity  of 

variance 

Bartlett's  chi  square  = 5,658  deg  freedom  = 4 p-value  = 0.226198 

The  variances  are  homogeneous  with  95%  confidence. 

If  samples  are  also  normally  distributed,  ANOVA  results  can  be  used. 

Kruskal-Wallis  One  Way  Analysis  of  Variance 

Kruskal-Wal lis  H (equivalent  to  Chi  square)  = 4.162 

Degrees  of  freedom  = 4 

p value  = 0,384587 
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Table  1.  Number  and  Percent  of  Participants  from  Each  Referral  Source 
(1952  Compared  to  1991) 


Referral  source 

1991 

1992 

family/friends 

9 (26%) 

8 (17%) 

legal  system 

4 (12%) 

12  (26%) 

school 

8 (24%) 

14  (30%) 

self 

7 (21%) 

7 (15%) 

other 

6 (18%) 

5 (11%) 

Figure  2.  Change  in  Number  of  Participants  From  Each  Referral  Source 

(1992  Compared  to  1991) 


_Q_  Self  ^ Other 
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Table  2.  Average  Age  of  Participants  from  Each  Referral  Source 


Referral  source 

Av^age  age 

family/ffiends 

17.4 

legal  system 

18.0 

school 

16.7 

self 

17.7 

other 

18.4 

/.  How  were  the  results  used? 

The  results  indicated  that  the  number  of  referrals  from  the  legal  system  and  from  schools  had 
increased  substantially  from  the  year  before,  without  any  loss  of  referrals  from  other  sources. 
Since  the  program  was  now  operating  at  capacity,  no  further  actions  were  planned  to  increase 
referrals.  The  program  manager  would  continue  to  communicate  with  all  referral  sources  on 
an  ongoing  basis  and  continue  to  monitor  referrals.  In  addition,  client  satisfaction  data  would 
be  analyzed  to  assess  the  extent  to  which  participants  from  different  referral  sources  found  the 
program  useful. 

The  data  on  age  of  participants  indicated  that  most  clients  were  in  the  16-19  age  group.  There 
seemed  to  be  no  major  differences  in  ages  of  participants  based  on  referral  source.  The  program 
would  continue  to  target  materials  to  this  age  group  and  continue  to  monitor  age  of  participants. 
In  addition,  client  satisfaction  responses  would  be  analyzed  to  assess  the  extent  to  which  the 
small  number  of  older  participants  found  the  program  useful. 


Methodological,  measurement  and  ethical  issues: 

Measurement:  The  categories  used  for  referral  sources  were  the  ones  most  frequently 
reported  on  the  intake  form.  The  "other"  category  included  those  referral  sources  that 
were  reported  too  infrequently  to  merit  a separate  category,  including  social  service 
agencies,  employer,  md  physicians. 
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Level  of  difficulty:  easy 

Level  of  resources  needed:  minimal 


Part  2.  Client  Satisfaction 


To  what  extent  did  participants  find  the  four  types  of  content  in  the 
program  useful? 

To  what  extent  did  participants  find  the  three  delivery  formats  of  the 
program  useful? 


a.  Who  is  asking  the  questions  and  why  do  they  want  this  information? 

On  an  ongoing  basis,  program  staff  need  to  know  if  clients  find  the  types  of  content  and  the 
delivery  formats  useful.  This  helps  staff  identify  when  changes  are  needed  in  the  program  and 
the  particular  aspects  of  the  program  that  need  to  be  changed. 


h.  What  resources  were  needed  to  collect  and  interpret  information? 

It  was  necessary  to  design  a short  questionnaire  to  be  completed  by  program  participants  (see 
final  page  of  this  part).  This  took  approximately  four  hours,  including  meeting  to  decide  on  the 
questions,  drafting  the  questions,  and  typing  and  duplicating  the  questionnaire.  It  also  took  the 
program  secretary  two  hours  to  enter  the  data  into  the  computer,  analyze  the  information  using 
EPI  INFO  and  prepare  summary  tables. 


c.  How  were  the  data  collected? 

At  the  end  of  the  fourth  session,  the  counsellor  distributed  questionnaires  to  program 
participants.  The  questionnaires  were  completed  anonymously.  Completed  questionnaires  were 
deposited  by  clients  in  a box  at  the  back  of  the  room  before  they  left  the  session. 

On  the  questionnaire,  program  participants  were  asked  to  rate  the  usefulness,  on  a five-point 
scale,  of  seven  different  elements  of  the  program.  The  content  for  each  of  the  four  sessions 
made  up  four  of  the  rating  categories,  and  the  manner  in  which  the  information  was  given  made 
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up  the  other  three  rating  categories.  Clients  were  asked  to  rate  the  usefulness  of  each  category 
on  a scale  of  1 - not  at  all  useful,  to  5 - very  useful. 

Pcuticipants  who  did  not  attend  the  fourth  session  (or  dropped  out  before  the  final  session)  were 
contacted  by  the  counsellor  in  order  to  find  out  why  they  did  not  attend.  These  clients  were 
asked  to  rate  program  components  for  the  sessions  they  attended,  if  any.  They  were  also  asked 
for  additional  comments. 


d.  How  were  the  data  analyzed? 

The  information  was  entered  and  analyzed  using  EPI  INFO  (see  Appendix  A).  Average  ratings 
for  each  of  the  program  components  were  calculated,  and  additional  comments  were  recorded. 
Figure  1 shows  the  EPI  INFO  commands  used  to  calculate  average  ratings. 

Lotus  1-2-3  was  used  to  prepare  a graphical  display  of  the  results. 


Figure  1.  Commands  and  Outputs  From  EIPI  INFO  Used  to  Compute  Mean  Ratings  for 
Program  Content  Delivery 


By  using  the  command,  MEANS  ALCOHOL,  the  following  table  is  generated: 


ALCOHOL  I Freq  Percent  Cum. 


1 I 1 2.2%  2.2% 

2 I 6 13.3%  15.6% 

3 I 12  26.7%  42.2% 

4 I 11  24.4%  66.7% 

5 I 15  33.3%  100.0% 


Total  I 45  100.0% 


Sum  = 168.00 
Mean  = Z.7Z 
Standard  deviation  = 1.14 


Similar  tables  are  generated  by  substituting  the  other  category  names  for  ALCOHOL  in  the 
MEANS  command  - e.g.,  MEANS  DRUGS,  MEANS  WHY  USE,  MEANS  HEALTHIER, 
MEANS  VIDEOS,  MEANS  DISCUSSION,  MEANS  HANDOUTS. 
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Forty-six  clients  attended  the  prevention  program  in  1992.  Seven  (15%)  dropped  out  before  the 
final  session  or  did  not  attend  the  final  session.  The  counsellor  was  able  to  contact  six  of  the 
seven  who  did  not  attend  the  final  session  and  obtain  ratings  on  some  aspects  of  the  program 
from  each  of  them. 

As  shown  in  Table  1,  all  four  types  of  session  contents  were  rated  useful,  with  "how  to  make 
healthier  choices"  rated  the  highest  (average  of  4.3  out  of  5)  and  "information  about  alcohol  and 
its  effects"  rated  the  lowest  (average  of  3.7  out  of  5).  The  delivery  formats  were  also  rated 
fairly  highly,  with  discussion  periods  rated  as  the  most  useful  of  the  three  formats.  These  results 
are  shown  in  bar  graph  format  in  Figure  2,  prepared  using  Lotus  1-2-3. 


Table  1.  Average  Usefulness  Rating  (l=not  at  all  useful  to  5= very  useful)  for 
Program  Content  and  Delivery 


Program  component 

Average  rating 

Types  of  content: 

information  about  alcohol  and  its  effects  (n=45) 

3.7 

information  about  other  drugs  and  their  effects  (n=42) 

4.2 

why  people  use  alcohol  and  other  drugs  (n=40) 

4.2 

how  to  make  healthier  choices  (n=39) 

4.3 

Delivery  format: 

videos  (n=39) 

3.9 

discussion  periods  (n=39) 

4.5 

handouts  (n=39) 

3.9 

Overall: 

4.0 
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Figure  2.  Average  Usefulness  Rating  for  Program  Content  TYP*  Delivery  Format 

Average  Overall  Rating  (out  of  5) 
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Some  of  the  suggestions  for  improving  the  program  were: 


• Use  more  videos. 

• Get  more  up-to-date  videos. 

• Do  it  all  in  one  week  instead  of  once  a week  for  four  weeks. 

• More  specific  information  about  drinking  and  driving  instead  of  general  information  about 
alcohol. 

• Not  so  many  facts  about  alcohol  and  other  drugs  - just  stuff  that  relates  to  us. 

• I knew  most  of  the  stuff  already. 

• Not  much  was  new  - find  out  what  we  already  know  before  you  tell  us  the  same  stuff 
again. 


Of  the  six  clients  who  did  not  complete  the  program  and  were  contacted  by  the  counsellor,  two 
indicated  that  they  did  not  want  to  come  and  no  one  could  force  them  to  do  so;  one  client  had 
obtained  a part-time  job  and  was  unable  to  attend  some  of  the  scheduled  sessions,  although  he 
indicated  that  he  had  found  the  sessions  that  he  did  attend  useful;  one  client  stated  that  her 
parents  were  no  longer  pressuring  her  to  attend,  and  she  indicated  that  she  "already  knew  it  all"; 
one  client  stated  that  she  had  no  way  of  getting  to  the  agency  for  the  sessions,  and  was  not 
particularly  interested  anyway;  and  one  client  stated  that  he  decided  he  did  not  feel  comfortable 
with  so  many  younger  participants.  In  the  last  instance,  the  client  was  21  years  of  age.  The 
counsellor  offered  to  present  the  four  sessions  to  him  individually,  but  the  client  declined,  stating 
that  perhaps  he  did  not  need  to  attend. 
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The  results  indicated  that  participants  generally  found  the  information  provided  by  the  program 
to  be  helpful.  However,  the  information  about  alcohol  and  its  effects  was  rated  least  helpful, 
and  some  of  the  suggestions  for  improvement  indicated  that  participants  would  have  preferred 
information  that  they  did  not  already  know.  Therefore,  program  staff  arranged  to  examine  the 
substance  abuse  curriculum  for  the  local  school  system  in  order  to  avoid  duplication  of 
information. 

Some  clients  indicated  that  they  would  like  to  have  information  that  is  more  relevant  to  them. 
To  identify  the  areas  that  participants  consider  to  be  most  relevant,  an  additional  section  was 
added  to  the  1993  questionnaire.  This  information  will  be  used  to  modify  the  program  the 
following  year. 

The  results  also  indicated  that  participants  found  the  delivery  formats  of  the  program  to  be 
useful.  However,  the  participants  indicated  that  the  videos  and  handouts  were  less  useful  than 
the  discussion  periods.  Suggestions  for  improvement  indicated  that  some  videos  seemed  dated. 
Program  staff  are  currently  viewing  new  videos  to  select  appropriate  ones  to  replace  those  that 
are  out-of-date. 

With  regards  to  the  use  of  handouts,  program  staff  are  reviewing  all  written  material  used  in  the 
program  to  select  handout  material  that  will  either  summarize  or  enhance  information  taught, 
and  not  simply  duplicate  the  information. 

Finally,  the  program  staff  implemented  a policy  where  no-shows  were  telephoned  immediately 
following  the  missed  session  in  order  to  identify  ways  that  the  program  could  facilitate 
attendance  (e.g.,  arranging  transportation,  if  needed). 
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Methodological,  measurement  and  ethical  issues: 

Methodological.  In  obtaining  client  feedback  it  is  important  to  contact  program  dropouts 
as  well  as  program  completers.  By  contacting  clients  who  did  not  return,  counsellors 
were  able  to  gain  a better  understanding  of  clients’  needs  (e.g.,  other  people  in  the  group 
too  young;  no  way  to  get  there)  and  of  why  clients  terminated.  Ratings  provided  only  by 
those  who  completed  the  program  tend  to  overestimate  the  usefulness  of  the  program  since 
clients  who  terminate  early  often  do  so  because  they  are  not  satisfied.  It  is  important  to 
gather  this  information  in  order  to  improve  the  program. 

Measurement.  A five-point  rating  scale  was  used  to  measure  client  ratings  of  usefulness 
in  order  to  gain  a better  understanding  of  the  degree  to  which  clients  found  the  services  to 
be  useful  or  not  useful.  A simple  yes/no  would  not  indicate  this  as  accurately.  As  well, 
the  rating  scales  were  presented  in  a neutral  format  so  as  not  to  bias  response.  Both  the 
rating  scale  and  the  use  of  non-judgmental  wording  in  the  questions  add  to  the  reliability 
of  the  responses  given. 
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Youth  Addiction  Services  Education  Program 

Feedback  Questionnaire 


Please  rate  how  useful  you  found  each  of  the  four  education  sessions: 

Check  here 

Circle  one  if  you  did 


not  at  all 
useful 

very 

useful 

not  attend 
that  session 

I. 

Alcohol  and  its  effects 

1 

2 

3 

4 

5 

□ 

11. 

Other  drugs  and  their  effects 

1 

2 

3 

4 

5 

□ 

III. 

Why  people  use 

1 

2 

3 

4 

5 

□ 

IV. 

Making  healthier  choices 

1 

2 

3 

4 

5 

□ 

Please  rate  how  useful  you  found  each  of  the  three  different  presentation  formats: 


not  at  all 
useful 

Circle  one 

very 

useful 

Videos 

1 

2 

3 

4 

5 

Group  discussion 

1 

2 

3 

4 

5 

Handouts 

1 

2 

3 

4 

5 

Gender:  Q male  Q female 

Who  referred  you  to  the  program?  (Check  one  only  - if  uncertain,  ask  counsellor.) 

1.  □ family  or  friends 

2.  □ legal  system 

3.  O school 

4.  □ self 

5.  O other 


THANK  YOU  FOR  HELPING  US  TO  IMPROVE  OUR  PROGRAM  ! 
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Level  of  difficulty:  easy 

Level  of  resources  needed:  moderate 


Part  3.  Client  Characteristics  by  Client  Satisfaction 

report  fitMng  the  progr^  useful? 

To  what  extent  did  pdtiicipdhts  fnM  dif^  sources  report  that 

the  progrant  was  useful? 

Were  older  particfpants  less  sdtiifiM  mth  the  pro  than  younger 
participdiits? t 


a.  Who  is  asking  the  questions  and  why  do  they  want  this  information? 

The  program  staff  felt  that  female  participants  were  much  more  receptive  and  participated  more 
in  the  sessions  than  did  male  participants.  If  certain  aspects  of  the  program  were  identified  as 
less  useful  for  males,  these  might  be  changed  to  better  address  the  needs  of  both  females  and 
males. 

Another  concern  related  to  the  usefulness  of  the  program  for  participants  from  different  referral 
sources.  With  the  increase  in  referrals  from  legal  sources  and  from  schools,  it  was  particularly 
important  to  find  out  whether  youth  from  these  referral  sources  found  the  program  useful. 

Finally,  some  of  the  open-ended  comments  on  the  satisfaction  questionnaire  suggested  that  older 
participants  did  not  find  the  program  as  useful  as  did  the  younger  participants. 


b.  What  resources  were  needed  to  collect  and  interpret  information? 

All  information  required  had  already  been  entered  on  the  computer;  therefore,  only  about  an 
hour  was  required  to  generate  average  scores  for  each  comparison  group  on  the  four  types  of 
content  and  the  three  delivery  formats  of  the  program  and  to  prepare  summary  tables. 
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c.  How  were  the  data  collected? 

All  required  information  had  been  collected  in  the  questionnaire  distributed  at  the  end  of  the 
fourth  session.  On  the  questionnaire,  program  participants  were  asked  to  rate  the  usefulness, 
on  a five-point  scale  (1  = not  at  all  useful  to  5 = very  useful). 

d.  How  were  the  data  analyzed? 

The  information  existed  in  a database  using  EPI  INFO  (see  Appendix  A).  The  commands 
required  to  generate  these  analyses  are  shown  in  Figure  1.  The  average  usefulness  ratings  of 
the  four  types  of  content  and  the  three  delivery  formats  were  computed  for  each  of  the  following 
comparisons:  gender,  referral  source,  and  age  group  (16  to  19  or  20-1- ). 

Graphical  comparisons  of  the  usefulness  ratings  were  prepared  with  Lotus  1-2-3  (see  Appendix 
B). 

Figure  1.  Commands  and  Outputs  From  EPI  INFO  Used  to  Compute  Mean  Ratings  for 
Each  of  the  Program  Content  Types  and  Delivery  Formats  by  Gender 

By  using  the  command,  MEANS  ALCOHOL  GENDER  /N,  the  following  statistics  are 
generated: 


MEANS 

of  ALCOHOL 

for  each 

category  of 

GENDER 

GENDER 

Obs 

Total 

Mean 

Variance 

Std  Dev 

1 

24 

83 

3.458 

1.216 

1.103 

2 

21 

85 

4.048 

1.248 

1.117 

Difference 

-0.589 

GENDER 

Minimum 

25%ile 

Median 

75%ile 

Maximum 

Mode 

1 

1.000 

3.000 

4.000 

4.000 

5.000 

4.000 

2 

2.000 

3.000 

5.000 

5.000 

5.000 

5.000 

ANOVA 

(For  normally  distributed  data  only) 

The  p value  is  equivalent  to  that  for  the  Student's  T Test, 
since  there  are  only  2 samples. 


Variation 

SS 

df 

MS 

F statistic 

p-value 

Between 

3.889 

1 

3.889 

3.161 

0.078945 

Within 

52.911 

43 

1.230 

Total 

56.800 

44 

Bartlett's  test  for  homogeneity  of  variance 
Bartlett's  chi  square  = 0.004  deg  freedom  = 1 p-value  = 0.952559 

The  variances  are  homogeneous  with  95%  confidence. 

If  samples  are  also  normally  distributed,  ANOVA  results  can  be  used. 

Mann-Whitney  or  Wilcoxon  Two-Sample  Test  (Kruskal-Wallis  test  for  two  groups) 

Kruskal-Wallis  H (equivalent  to  Chi  square)  = 3.139 

Degrees  of  freedom  = 1 

p value  = 0.076433 

MEANS  ALCOHOL  GENDER  /N 

Average  ratings  can  be  calculated  for  referral  source  and  age  group  by  substituting  Referral  or  AgeGrp 
for  GENDER.  Similarly,  other  content  areas  and  formats  can  be  substituted  for  the  variable  ALCOHOL. 
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Forty-six  clients  attended  the  prevention  program  in  1992.  Forty-five  (97.8%)  completed  at  least 
one  program  usefulness  rating  on  the  client  feedback  sheet.  Thirty-nine  (84%)  provided 
responses  to  all  seven  usefulness  ratings  on  the  client  feedback  questionnaire.  The  analyses  that 
follow  were  based  on  data  from  those  participants  who  provided  usefulness  ratings  on  all  seven 
program  components. 

As  shown  in  Table  1,  female  participants  tended  to  rate  the  program  more  highly  than  male 
participants.  The  same  results  are  shown  in  a line  graph  in  Figure  2,  produced  using  Lotus 
1-2-3. 


Table  1.  Average  Usefulness  Rating  for  Program  Content  and  Delivery  (Female 
Compared  to  Male  Participants) 


Female 

Male 

Program  component  ^ 

n=18 

n-21 

alcohol  and  its  effects 

4.0 

3.5 

other  drugs  and  their  effects 

4.4 

3.9 

why  people  use 

4.6 

3.9 

making  healthier  choices 

4.7 

4.0 

videos 

4.4 

3.4 

discussion 

4.8 

4.2 

handouts 

4.3 

3.5 

overall 

4.4 

3.6 
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(Female  Compared  to  Male  Participants) 


, Female  » Male 


Some  differences  emerged  on  overall  satisfaction  levels  of  participants  from  different  referral 
sources,  with  legal  referrals  giving  the  lowest  ratings  (see  Table  2 and  Figure  3). 


Table  2.  Average  Usefulness  Rating  for  Program  Content  and  Delivery  (Comparison 
Among  Participants  From  Elach  Referral  Source) 


Program  component 

Family/friends 

n=5 

Legal 

n=ll 

School 

n=10 

Self 

n=7 

Other 

n=5 

alcohol  and  its  effects 

4.1 

3.0 

3.7 

3.7 

4.8 

other  drugs  and  their  effects 

4.4 

3.1 

4.5 

4.4 

4.8 

why  people  use 

4.7 

3.1 

4.5 

4.4 

5.0 

making  healthier  choices 

4.8 

3.3 

4.7 

4.6 

5.0 

videos 

4.3 

2.9 

4.3 

4.0 

4.4 

discussion 

4.8 

3.9 

4.6 

4.6 

4.8 

handouts 

4.5 

2.8 

3.8 

4.3 

4.8 

overall 

4.5 

3.1 

4.4 

4.0 

4.6 
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Figure  3.  Average  Usefulness  Rating  for  Program  Content  and  Delivery 
(Comparison  Among  Participants  From  Each  Referral  Source) 


, Family/friends  » Legal  ,.4_  School 

□ Self  ^ Other 


As  shown  in  Table  3 and  Figure  4,  younger  participants  tended  to  rate  the  program  more  useful 
than  older  participants;  however,  since  only  four  participants  were  over  the  age  of  19,  these 
results  need  to  be  interpreted  with  caution. 
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Table  3.  Average  Usefulness  Ratings  for  Program  Content  and  Delivery  Format 
(Participants  Aged  16  to  19  Compared  to  Those  Aged  20  and  Older) 


Program  component 

Aged  16-19  years 
n=35 

Aged  20+  years 
n=4 

alcohol  and  its  effects 

3.8 

3.3 

other  drugs  and  their  effects 

4.3 

3.2 

why  people  use 

4.3 

3.4 

making  healthier  choices 

4.4 

3.6 

videos 

4.0 

3.0 

discussion 

4.6 

3.8 

handouts 

3.9 

3.4 

overall 

4.1 

3.2 

Figure  4.  Average  Usefulness  Rating  for  Program  Content  and  Delivery 
(Participants  Aged  16  to  19  Compared  to  Those  Aged  20  or  Older) 
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The  results  indicated  that  females  consistently  rated  the  program  elements  more  positively  than 
did  males.  This  seemed  to  be  a general  trend,  rather  than  related  to  specific  course  content. 
The  results  provided  no  clear  direction  for  revising  the  program  to  make  it  more  satisfactory  for 
males.  A decision  was  made  to  continue  to  monitor  this  trend. 

Although  the  policy  set  up  with  the  legal  system  for  referrals  of  high-risk  youth  to  Youth 
Addiction  Services  increased  referrals  to  the  program  in  1992,  the  results  indicated  that  these 
individuals  did  not  rate  the  program  as  very  useful.  This  could  mean  that  the  types  of  content 
and  the  delivery  formats  need  to  be  re-examined  for  legal  referrals.  On  the  other  hand,  it  may 
be  that  these  individuals,  perhaps  as  a result  of  being  forced  to  attend  the  program,  resisted 
participating.  The  program  staff  presented  this  result  to  the  community  advisory  committee  and 
recommended  that  alternate  arrangements  be  considered  for  addressing  the  needs  of  the  legal 
referrals.  The  advisory  committee  felt  that  it  was  important  to  continue  to  monitor  the  suitability 
of  the  community  program  for  legal  referrals  before  making  changes.  Given  that  the  new 
questionnaire  will  collect  additional  information,  evaluations  for  1993  will  be  examined  closely 
to  determine  possible  changes  that  might  increase  satisfaction  levels  among  legal  referrals. 

Although  participants  who  were  over  19  years  of  age  tended  to  rate  the  program  less  highly  than 
younger  participants,  there  were  only  four  in  the  older  group  who  participated  in  the  feedback 
survey.  This  finding  may  indicate  that  the  program  is  less  useful  for  older  participants,  but 
more  data  need  to  be  collected  to  adequately  assess  age  effects. 


Methodological,  measurement  and  ethical  issues: 

Methodological.  Comparing  the  satisfaction  levels  of  participants  from  different  referral 
sources  for  only  one  year  involves  average  scores  calculated  on  samples  as  small  as  four 
people.  Therefore,  the  reliability  of  observed  differences  among  referral  sources  is 
questionable.  With  such  small  numbers,  it  is  important  to  monitor  differences  over  several 
years  so  that  conclusions  Ccm  be  based  on  larger  samples  from  each  referral  source. 

Although  the  questionnaires  contained  each  participant’s  age,  gender,  and  referral  source, 
names  were  not  included.  In  collecting  client  satisfaction  data  it  is  important  that  clients  be 
able  to  respond  anonymously. 
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Program:  The  program  is  an  outreach  program  for  older  persons  who  have  problems  related 
to  alcohol  or  other  drugs.  The  program  provides  a broad  range  of  case  management  services, 
including  assessment,  supportive  counselling,  crisis  intervention  and  advocacy.  Clients  are  seen 
in  their  homes  or  at  the  program  office  depending  on  their  needs.  Duration  of  contact  with  the 
program  can  range  from  one  or  two  sessions  to  several  years. 

This  example  includes  quality  enhancement  questions  for  four  levels  of  evaluation: 


Part  1.  Client  Characteristics 

Part  2.  Program  Delivery 

Part  3.  Outcome  Evaluation 

Part  4.  Client  Characteristics  by  Program  Delivery. 
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Level  of  difficulty:  easy 

Level  of  resources  needed:  minimal 


Part  1.  Client  Characteristics 


How  many  clients  were  experiencing  accommodation  problems? 


a.  Who  was  asking  the  question  and  why  did  they  want  this  information? 

The  case  workers  in  the  program  had  observed  that  they  were  spending  a significant  amount  of 
time  finding  housing  for  clients.  Sometimes  the  housing  problem  was  related  to  the  client’s 
alcohol  or  drug  problem  (e.g.,  the  client  was  being  evicted  for  drunken  behaviour);  sometimes 
the  problem  was  finding  affordable  housing  for  a person  on  a very  low  income;  and  sometimes 
supportive  housing,  such  as  a nursing  home,  was  needed  because  the  client  was  unable  to  care 
for  him  or  herself  (because  of  aging,  substance  abuse  or  both). 

Finding  suitable  housing  for  some  clients  was  very  difficult.  Many  places  did  not  want  someone 
with  an  active  alcohol  problem.  There  was  also  a need  for  housing  that  provided  some  support 
but  still  allowed  considerable  independent  living.  For  example,  some  clients  were  not  ill  enough 
to  move  to  a nursing  home  but  not  really  well  enough  to  live  entirely  on  their  own. 

The  program  manager  and  staff  wanted  information  on  their  clients’  housing  problems  in  order 
to  have  strong  evidence  to  lobby  with  municipal  and  provincial  decision-makers  regarding  the 
range  of  housing  support  needs  for  older  people.  The  program  was  also  interested  in  exploring 
whether  there  was  sufficient  need  to  open  a supportive  home  specifically  for  older  alcohol  and 
drug  abusers. 


h.  What  resources  were  needed  to  collect  and  interpret  information? 

Information  regarding  accommodation  problems  was  collected  as  part  of  initial  assessment.  In 
addition,  accommodation  problems  that  emerged  during  treatment  were  recorded.  A case  worker 
spent  six  hours  reviewing  client  files,  recording  the  number  of  clients  who  experienced 
accommodation  problems,  and  recording  the  nature  of  problems  experienced. 
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c.  How  were  the  data  collected? 

The  program  record  keeping  included  a comprehensive  assessment  (where  a number  of  questions 
were  asked  about  accommodation  problems).  In  addition,  program  workers  used  a brief  form 
to  keep  a record  of  problems  that  emerged  during  the  treatment  process.  Data  on  all  cases 
admitted  during  the  previous  year  were  examined  to  identify:  (a)  the  proportion  of  all  clients 
who  experienced  accommodation  problems  and  (b)  the  nature  of  accommodation  problems 
experienced. 


d.  How  were  the  data  analyzed? 

The  information  was  tallied  by  hand. 

A pie  graph  showing  the  proportion  of  clients  with  different  accommodation  problems  was 
generated  using  Lotus  1-2-3  (see  Appendix  B). 


e.  What  did  they  find  out? 

Of  75  cases,  25  (33%)  were  found  to  have  some  level  of  accommodation  problems.  Table  1 
summarizes  the  nature  of  accommodation  problems  experienced.  These  results  are  shown  in  pie 
chart  form  in  Figure  1 (prepared  using  Lotus  1-2-3). 


Table  1.  Types  of  Accommodation  Problems  Identified 


Type  of  accommodation  problem 

Number  and  % who 
experienced  it. 

cognitive  or  physical  impairment  requiring  move  to  long-term  care 

5 (7%) 

accommodation  problems  related  to  alcohol  use  that  disappeared 

10  (13%) 

when  the  client  reduced  or  quit  drinking 

able  to  live  independently  but  needing  some  support  such  as  Meals- 

10  (13%) 

on-Wheels  and  homemaking  services 

Of  the  10  who  needed  support  but  could  maintain  fairly  independent  living,  four  never  obtained 
stable  housing  during  the  time  period  of  the  evaluation.  For  the  remaining  six,  supports  were 
arranged  using  a good  boarding  home  or  friends  or  relatives,  as  well  as  community  services  such 
as  Meals-on-Wheels. 
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Figure  1.  Types  of  Accommodation  Problems  Identified  by  Clients 


/.  How  were  the  results  used? 

Of  those  experiencing  accommodation  problems,  10  required  institutionalization  in  a long-term 
care  facility  and  five  eventually  achieved  totally  independent  living.  Some  of  the  remaining  10 
might  have  benefited  from  the  availability  of  a supportive  residence  for  older  alcohol  and  drug 
abusers.  This  number  did  not  seem  to  justify  opening  such  a residence  at  this  time,  but  the 
information  compiled  in  this  evaluation  was  used  to  influence  legislation  regarding  community 
support.  In  addition,  the  information  was  helpful  in  training  new  workers  regarding  the  nature 
of  housing  problems  that  occurred  and  the  kinds  of  solutions  that  had  been  found  for  previous 
clients. 
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Level  of  difficulty:  easy 

Level  of  resources  needed:  moderate 


Part  2.  Program  Delivery 


How  many  Jiours  of  service  were  provided  to  each  client? 

What  proportion  of  these  hours  were  contact  hours  with  the  client 
compared  to  the  proportion  of  hours  involving  contact  with  others  about 
the  client? 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 

The  program  board  requested  that  the  program  begin  to  compile  some  basic  statistics  on  hours 
of  direct  service.  This  basic  information  was  needed  to  identify  the  types  of  clients  who  were 
likely  to  require  more  time.  In  particular,  the  board  and  the  program  staff  wished  to  explore 
whether  clients  who  had  accommodation  problems  needed  more  service.  In  order  to  address  this 
and  other  questions  in  the  future,  a system  of  monitoring  contact  hours  was  needed. 


h.  What  resources  were  needed  to  collect  and  interpret  the  information? 

The  data  collection  required  an  extra  10-20  minutes  per  week  for  each  of  two  case  workers  to 
complete  weekly  logs  for  average  caseloads  of  20  clients  (a  total  of  75  clients  for  the  year). 
This  produced  about  1,500  logs  (since  not  all  clients  on  the  active  caseload  received  services 
every  week).  Inputting  the  data  on  the  computer  and  tabulating  the  results  required  about  20 
hours  of  the  secretary’s  time. 


c.  How  were  the  data  collected? 

For  one  year,  the  program  workers  kept  a log  of  all  client-related  activities.  The  log  included: 
(a)  who  the  contact  was  with  and  (b)  the  length  of  the  contact  in  minutes. 
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The  information  was  entered  into  the  computer  using  Lotus  1-2-3  (see  Appendix  B).  The 
secretary  calculated  the  average  number  of  hours  of  service  for  client  contacts  and  for  contacts 
with  others  about  specific  clients. 


e.  What  did  they  find  out? 

Table  1 shows  how  contact  hours  regarding  specific  clients  during  the  previous  year  were 
distributed  between  time  spent  directly  with  clients  and  time  spent  with  others  regarding  clients. 
Others  involved  in  contacts  included  family  members,  friends  of  the  client,  and  other  agencies 
involved  with  the  client. 


Table  1.  Average  Number  of  Contact  Hours  With  Clients  and  With  Others  About 
Clients 


Average  number 
of  hours  per  client 

average  number  of  total  contact  hours  per  client 

10.2 

average  number  of  contact  hours  with  the  client  only 

5.8 

average  number  of  contact  hours  with  both  the  client  and  someone  else 

2.4 

average  number  of  contact  hours  with  someone  else  about  the  client 

2.0 

/.  How  were  the  results  used? 

The  information  collected  helped  the  board  and  the  program  manager  to  identify  the  relative 
amounts  of  time  spent  with  clients  compcU’ed  to  the  time  spent  with  others  concerning  the  clients. 
This  information  was  also  used  to  address  more  complex  evaluation  questions,  such  as 
identifying  the  types  of  clients  who  required  more  services  (see  Part  4 of  this  example). 
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Level  of  difficulty:  easy 

Level  of  resources  needed:  moderate 


Part  3.  Outcome  Evaluation 


How  many  clients  were  doing  well  12  months  after  discharge  from 
treatment? 


a.  Who  was  asking  the  question  and  why  did  they  want  this  information? 

The  board  asked  for  some  statistics  on  how  many  people  were  succeeding  in  the  program.  The 
program  had  been  in  operation  for  three  years  and  the  board  felt  that  it  was  time  to  begin 
keeping  basic  statistics  on  long-term  outcomes. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

Most  of  the  information  required  had  previously  been  collected  from  clients  contacted  for  follow- 
up. About  10  hours  were  needed  to  track  down  information  on  the  15  clients  who  had  not  been 
available  for  follow-up.  An  additional  three  hours  were  spent  compiling  the  results. 

c.  How  were  the  data  collected? 

Part  of  the  program’s  services  included  a clinical  follow-up  with  clients  at  six  and  12  months 
following  discharge  to  obtain  feedback  about  the  program  and  to  provide  additional  support,  if 
needed.  To  address  the  evaluation  question,  the  case  workers  also  tried  to  obtain  outcome 
information  on  those  clients  who  had  not  been  contacted  at  follow-up.  This  extra  information 
was  obtained  by  exploring  the  following  sources:  the  initial  referring  agent  (e.g. , a public  health 
nurse),  the  client’s  family,  the  client’s  last  known  address,  death  records,  and  asking  others  who 
knew  the  client. 

The  case  workers  usually  tried  to  contact  their  own  clients  as  they  felt  this  would  be  most 
comfortable  for  the  client.  However,  since  one  case  worker  who  had  been  involved  in  providing 
treatment  to  the  clients  who  were  being  followed  up  had  since  left  the  program,  some  clients 
were  contacted  by  a case  worker  whom  they  did  not  know. 
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Usually  the  case  worker  spoke  with  the  client  by  telephone.  A few  of  the  clients  dropped  by 
the  office  to  chat  occasionally  and  follow-up  data  were  collected  from  them  during  one  of  their 
visits.  Several  clients  had  cognitive  or  other  problems  so  that  the  follow-up  needed  to  be  done 
at  their  residence.  Each  person  was  asked  about  their  alcohol  and  drug  use  during  the  past  12 
months.  They  were  also  asked  how  they  were  doing  in  other  areas  of  their  life  and  whether  they 
needed  further  assistance  from  the  program. 


d.  How  were  the  data  analyzed? 

The  information  on  outcomes  was  tallied  by  hand  by  the  program  secretary.  Status  at  follow-up 
was  compared  to  the  clients’  status  when  they  were  discharged  from  the  program. 

Graphical  presentations  of  the  results  were  generated  using  Lotus  1-2-3  (see  Appendix  B). 


e.  What  did  they  find  out? 

The  case  workers  were  able  to  obtain  information  on  36  of  the  40  clients  who  were  eligible  for 
a 12-month  follow-up  interview  during  the  preceding  year.  As  shown  in  Table  1,  clients  doing 
well  at  discharge  tended  to  continue  to  do  well  at  follow-up;  those  doing  poorly  continued  to  do 
poorly.  In  all,  20  (56%)  were  rated  as  "good"  status  at  follow-up  (17  who  had  been  rated 
"good"  at  discharge  and  three  who  had  been  rated  "no  improvement"  at  discharge).  Not  all  had 
been  abstinent  or  even  problem-free,  but  all  20  reported  that  they  were  generally  doing  much 
better  than  before  treatment  and  that  problems  related  to  substance  abuse  were  greatly  reduced 
or  eliminated.  Four  (11%)  appeared  to  be  unimproved.  The  remaining  12  (33%)  had  either 
died  or  their  condition  had  deteriorated.  It  was  not  possible  to  assess  the  extent  to  which  death 
or  deterioration  was  directly  related  to  post-program  alcohol  or  drug  abuse.  It  was  known, 
however,  that  in  at  least  some  instances,  these  outcomes  were  related  to  the  normal  events  of 
aging  (possibly  related  to  long-term  substance  abuse),  but  not  related  to  post-treatment  substance 
use.  For  example,  one  person  had  been  hospitalized  for  cancer  at  the  time  of  discharge  and  died 
without  ever  returning  to  drinking.  Figures  1,  2 and  3 (produced  using  Lotus  1-2-3)  are  pie 
charts  which  show  the  proportion  of  clients  whose  status  at  follow-up  was  classified  as:  good, 
no  improvement,  or  poor  for  each  level  of  status  at  discharge. 
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Table  1.  Status  at  Discharge  by  Status  at  Follow-Up 


Status  at  jPbUow-up 

Status  at 
discharge 

Good 

No  improvident 
from  pre-treats 
level 

Poor 

good 

5 abstinent  - doing  well 

3 occasional  light  drinking 

- doing  well 

4 occasional  heavy  drinking 

- doing  well 

3 abstinent  - but  other  health  or 
emotional  problems 

1 apparently  no  problems 
related  to  drinking 
(consumption  level  not 
reported) 

1 abstinent  from  alcohol  - but 
one  bout  with  drug  addiction 
since  discharge 

1 would  not  answer 
questions,  but  known 
to  have  had  at  least 
one  major  relapse 

1 abstinent  at  follow- 
up, but  had 
experienced  severe 
psychiatric  problems 
as  well  as  one  major 
relapse  since 
discharge 

3 deceased 

no 

improvement 

1 abstinent  - doing  well 

1 occasional  heavy  drinking 
- doing  well 

1 apparently  no  problems 
related  to  drinking 

1 would  not  report 
consumption  level, 
but  believed  to  be 
drinking  heavily 

1 binges  and  heavy 
drinking 

5 deceased 

1 institutionalized 

poor 

1 institutionalized 

1 living  with  family,  but 
too  cognitively 
impaired  to  be 
interviewed;  still 
drinking 

1 drinking  5 drinks  per 
day 
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Fieure  1 Status  of  Clients  at  Discharge  by  Status  at  12-Month  Follow-up:  Figure  2.  Status  of  Clients  at  Discharge  by  Status  at  12-Montb  Follow-up: 

S Good  status  at  discharge  No  improvement  status  at  discharge 


/.  How  were  the  results  used? 
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The  information  was  collected  in  order  to  begin  a process  of  monitoring  12-month  outcomes  as 
part  of  the  collection  of  basic  statistics.  It  was  hoped  that  once  data  had  been  collected  for 
several  years,  sufficient  data  would  be  available  to  identify  those  clients  who  seemed  to  respond 
best  to  the  program.  As  well,  they  hoped  to  identify  clients  who  did  not  respond  to  the  program 
in  order  to  develop  better  ways  of  reaching  these  people. 


Methodological,  measurement  and  ethical  issues: 

Methodological.  Although  finding  clients  of  addictions  treatment  programs  12  months 
after  treatment  tends  to  be  difficult  for  most  programs,  older  people  are  less  likely  to  move 
and  are  more  easily  tracked  through  the  health  system.  Thus,  this  evaluation  was  able  to 
achieve  a high  rate  of  follow-up  information  (on  36  out  of  40  cases). 

Measurement.  Outcome  status  was  assessed  using  mostly  self-report  by  the  client 
(although  in  some  instances,  the  case  worker  was  able  to  confirm  the  status  through  other 
sources).  There  may  have  been  some  respondents  who  did  not  want  to  report  substance 
abuse  problems  because  they  were  embarrassed,  or  did  not  want  to  hurt  the  case  worker’s 
feelings,  or  did  not  want  to  cause  problems  for  the  program.  In  each  interview,  the  case 
workers  stressed  that  they  really  wanted  the  client  to  be  honest  and  that  the  program  really 
wanted  the  outcome  statistics  to  be  accurate.  Nevertheless,  the  results  should  be  interpreted 
cautiously,  as  they  may  be  positively  biased. 

Another  source  that  could  bias  the  results  in  a positive  direction  is  that  case  workers  are  not 
objective  regarding  the  outcome.  The  case  workers  who  contacted  their  own  clients  would 
hope  for  the  clients  to  be  doing  well  for  the  client’s  sake  and  also  to  feel  that  they  had  done 
a good  job.  It  was  felt  by  the  program  manager  that  the  clinical  value  of  having  the  client 
contacted  by  his  or  her  own  caseworker,  when  possible,  superseded  the  risk  of  bias.  To 
reduce  some  of  the  bias,  the  manager  assured  the  case  workers  that  the  results  would  not  be 
analyzed  separately  for  each  worker  and  the  manager  intentionally  would  not  ascertain  the 
success  rates  of  individual  workers. 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  moderate 


Part  4.  Client  Characteristics  by  Program  Delivery 


Did  clients  who  had  accommodation  problems  receive  more  hours  of 
service  overall  than  those  who  did  not  have  accommodation  problems? 

Did  clients  who  had  accommodation  problems  require  more  hours  of 
service  with  other  people  on  their  behdlf? 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 

The  case  workers  in  the  program  had  the  perception  that  resolving  accommodation  problems  for 
clients  was  taking  a lot  of  their  time.  For  clients  who  had  accommodation  problems,  the  case 
worker  had  to  spend  considerable  time  problem-solving  with  the  client,  advocating  or  interacting 
in  other  ways  with  owners  of  rental  property,  investigating  various  options  for  subsidized 
housing,  involving  family  members  in  trying  to  find  a solution,  and  engaging  in  other  activities 
directed  towards  resolving  accommodation  problems.  The  program  manager  and  staff  (with 
the  support  of  the  board)  decided  to  investigate  this  issue.  The  first  step  in  the  process  was  to 
identify  the  proportion  of  clients  who  had  problems  in  accommodation  (described  in  Part  1 of 
this  example).  The  next  step  was  to  estimate  the  extra  time  involved  in  resolving 
accommodation  problems.  They  did  not  have  exact  information  with  respect  to  the  time  spent 
directly  on  accommodation  problems,  since  a contact  with  a client  or  family  member  might 
include  problem  solving  about  accommodation  as  well  as  the  discussion  of  several  other  issues. 
Therefore,  they  framed  a question  that  could  provide  a partial  answer  to  their  concern  - namely, 
did  clients  who  had  accommodation  problems  receive  more  services  than  those  who  did  not? 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

The  rating  of  clients  as  to  whether  or  not  they  had  accommodation  problems  had  been  collected 
during  assessment.  These  data  as  well  as  data  regarding  hours  of  service  had  already  been 
entered  into  the  computer.  The  program  secretary  spent  about  two  hours  extracting  and 
summarizing  the  information  required  to  address  the  questions  used  in  this  part  of  the  evaluation. 
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c.  How  were  the  data  collected? 

Addressing  the  question  required  examining  data  already  collected  in  Parts  1 and  2.  The  logs 
of  client-related  activities  had  allowed  the  program  to  estimate  the  average  number  of  hours  of 
direct  service  provided  to  the  client  as  well  as  hours  of  service  to  others  about  the  clients. 


d.  How  were  the  data  analyzed? 

The  data  from  the  logs  of  client-related  activities  had  already  been  entered  into  Lotus  1-2-3. 
These  logs  were  then  divided  into  those  clients  who  had  accommodation  problems  and  those  who 
did  not.  The  average  number  of  hours  of  service  for  client  contacts  and  for  contacts  with  others 
about  the  client  were  calculated  for  the  two  groups. 


e.  What  did  they  find  out? 

Of  75  clients  assessed,  25  had  identified  accommodation  problems.  Table  1 shows  the  average 
number  of  contact  hours  during  the  year  of  the  study  for  clients  who  had  accommodation 
problems  compared  to  those  who  did  not  have  accommodation  problems.  In  general,  those 
clients  who  had  accommodation  problems  tended  to  require  more  hours  of  service  than  those 
who  did  not  have  accommodation  problems.  The  same  results  are  shown  in  a bar  graph  form 
in  Figure  1. 
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Table  1.  Average  Number  of  Contact  Hours  for  Clients  Who  Had  Accommodation 
Problems  Compared  to  Clients  Who  Did  Not  Have  Accommodation  Problems 


Contact  hours  for  clients 
who  had  accommodation 
problems 
n=25 

Contact  hours  for  clients 
who  did  not  have 
accommodation  problems 
n=50 

average  number  of  total  contact  hours 
per  client 

12.3 

9.2 

average  number  of  contact  hours  with 
the  client  only 

6.0 

5.7 

average  number  of  contact  hours  with 
both  the  client  and  someone  else 

3.1 

2.1 

average  number  of  contact  hours  with 
someone  else  about  the  client 

3.2 

1.4 

Figure  1.  Average  Number  of  Contact  Hours  (Clients  Who  Had  Accommodation  Problems 
Compared  to  Clients  Who  Did  Not  Have  Accommodation  Problems) 
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The  results  confirmed  the  perception  that  those  who  had  accommodation  problems  received  more 
services.  In  particular,  the  workers  spent  more  time  meeting  with  others  on  the  client’s  behalf 
for  clients  who  had  accommodation  problems.  The  average  number  of  hours  of  direct  service 
to  the  client  alone,  however,  was  similar  regardless  of  whether  or  not  the  clients  had 
accommodation  problems. 

The  results  had  at  least  two  implications  for  the  program.  First,  if  the  program  continued  to  see 
a fairly  large  proportion  of  clients  who  had  accommodation  problems,  it  might  be  cost-effective 
to  spend  time  further  exploring  accommodation  options  in  the  community  and  lobbying  for  better 
or  more  appropriate  options. 

Second,  if  no  changes  occurred  in  the  way  cases  were  handled,  program  planning  should 
incorporate  the  expectation  that  clients  who  had  accommodation  problems  were  more  likely  to 
need  extra  hours  of  service. 


Methodological,  measurement  and  ethical  issues: 

Methodological.  Although  the  data  showed  that  those  who  had  accommodation  problems 
required  more  services,  it  is  only  an  assumption  that  these  extra  services  are  because  of  the 
accommodation  problems.  An  alternative  explanation  is  that  those  who  have 
accommodation  problems  are  more  likely  to  be  skid  row  type  drinkers,  and  therefore  have 
many  lifestyle  problems  that  need  to  be  addressed.  It  is  possible  that  the  extra  hours  of 
service  are  because  of  the  deteriorated  state  of  the  client  rather  than  the  accommodation 
problems,  per  se. 

Measurement.  Measurement  was  kept  fairly  simple  to  allow  the  program  to  accomplish  a 
quick  examination  of  the  relationship  between  hours  of  service  and  accommodation 
problems.  Clients  were  only  categorized  into  two  categories:  problem/no  problem.  The 
findings  might  be  more  informative  if  a severity  scale  was  used  to  rate  accommodation 
problems  and  if  the  nature  of  accommodation  problems  was  included  in  the  analyses. 

On  the  other  hand,  this  fairly  simple  evaluation  allowed  the  program  to  address  an  issue  of 
interest.  Since  the  findings  indicated  a difference  in  level  of  service  to  those  with 
accommodation  problems,  the  program  might  choose  to  monitor  accommodation  problems 
in  more  detail  in  the  future  in  order  to  identify  the  specific  aspects  of  treatment  that  were 
most  time-consuming. 
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J.  Detox  Needs  Assessment  and  Evaluation 


Program:  A new  detox  program  was  set  up  to  provide  non-medical  detox  services  for  both 
men  and  women  in  the  county.  The  program  has  16  beds,  two  of  which  are  allotted  to 
individuals  from  outside  the  county.  The  average  length  of  stay  at  the  unit  is  three  days. 
Individuals  are  generally  admitted  in  order  to  detoxify,  and  assessments  for  subsequent  treatment 
are  sometimes  completed  while  they  are  at  the  detox. 

The  needs  assessment  (Part  1)  was  completed  a number  of  years  ago,  and  the  detox  program 
proposal  accepted  two  years  ago.  The  unit  received  funding  from  the  Ministry  of  Health  one 
year  ago. 

This  example  includes  quality  enhancement  questions  for  three  levels  of  evaluations,  including 
the  needs  assessment  that  was  done  originally  in  planning  for  the  program: 


Part  1. 

Needs  Assessment 

Part  2. 

Client  Characteristics 

Part  3. 

Program  Delivery. 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  moderate 


Part  1.  Needs  Assessment 


Were  some  clients  of  the  local  assessment/referral  and  day  treatment 
programs  who  appeared  to  need  detoxification  unable  to  obtain  detox 
services  because  of  problems  associated  with  the  location  of  outlying  detox 
programs? 

What  were  the  polimes  of  detox  programs  in  adjacent  counties  regarding 
referrals  from  outside  their  catchment  areas  ? 

To  what  extent  did  Clients  who  attended  detox  programs  in  other  counties 
experienceproblems  related  to  the  distant  location  of  the  detox  programs? 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 

A county- wide  substance  abuse  task  force  comprised  of  service  providers,  concerned  citizens, 
staff  of  the  county  District  Health  Council  (DHC)  and  addictions  consultants  organized  the  needs 
assessment  to  evaluate  the  need  for  a local  detox  program. 

Although  there  were  detox  services  available  in  two  adjacent  counties,  there  was  some  concern 
that  these  services  were  not  adequate.  First,  the  distance  to  these  programs  posed  transportation 
or  other  problems  for  potential  users.  Second,  these  services  primarily  served  their  own 
counties  and  had  limited  space  for  persons  from  other  counties.  Finally,  it  appeared  that  those 
who  obtained  detox  services  outside  of  the  community  were  not  linked  to  local  treatment 
programs  as  often  as  believed  optimal  by  the  task  force. 

The  local  assessment/referral  program  and  day  treatment  program  agreed  to  help  in  analyzing 
the  detox  needs  of  the  clients  they  had  seen  during  the  previous  two  years.  While  this  did  not 
allow  for  a description  of  all  potential  users  of  detox  services,  it  provided  at  least  some  data  that 
could  be  used  to  assess  local  need. 
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b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

A university  student  was  hired  by  the  DHC  on  a full-time  basis  for  the  summer  (four  months). 
The  job  entailed  designing  the  evaluation,  constructing  and  coding  questionnaires,  gathering  data, 
entering  the  data  in  the  computer,  completing  the  data  analyses,  and  generating  a report.  As 
well,  one  staff  member  of  the  DHC  spent  40  hours  training  and  supervising  the  student. 

In  addition  to  the  student  and  DHC  staff  time,  the  program  managers  of  the  county 
assessment/referral  and  day  treatment  programs  had  agreed  to  assist  the  student  in  gathering 
information  from  previous  clients.  The  secretary  of  the  assessment/referral  centre  spent  15 
hours  retrieving  information  from  a computer  database,  reviewing  case  files,  generating  a 
mailing  list,  and  mailing  out  the  questionnaires  provided  by  the  student.  The  program 
administrator  of  the  day  treatment  program  spent  six  hours  reviewing  hand-tabulated 
information,  and  the  secretary  spent  two  hours  generating  a mailing  list  and  mailing  out 
questionnaires. 


c.  How  was  the  information  collected? 

The  first  task  was  to  determine  how  many  clients  of  the  assessment/referral  and  day  programs 
had  been  identified  as  needing  detox,  and,  of  these,  how  many  actually  went  to  a detox  program. 
Client  records  over  the  previous  two-year  period  were  reviewed  to  determine  who  had  been 
referred  to  out-of-county  detox  services,  as  well  as  the  proportion  of  clients  for  whom  detox 
arrangements  could  not  be  made  despite  an  apparent  need.  Reasons  were  noted  for  those  clients 
who  had  been  recommended  to  use  detox  services  but  did  not  use  them. 

The  second  task  involved  sending  questionnaires  to  clients  of  the  assessment/referral  program 
or  the  day  program  who  used  an  out-of-county  detox  program.  In  the  questionnaire,  respondents 
were  asked  to  rate  a number  of  potential  problems  resulting  from  the  need  to  use  out-of-county 
detoxes  on  a scale  of  1 to  3:  (1)  Not  at  all  a problem;  (2)  Somewhat  of  a problem;  and  (3)  Very 
much  a problem.  Additional  comments  were  also  encouraged  (see  questionnaire  at  end  of  this 
section). 

The  questionnaires  were  mailed  by  the  assessment/referral  and  day  treatment  programs  to  those 
clients  who  had  used  out-of-county  detox  services.  The  secretaries  of  the  two  programs 
addressed  the  questionnaires  and  mailed  them  to  relevant  clients  (along  with  a self-addressed 
stamped  envelope  for  returning  the  completed  questionnaire  to  the  student).  Because  the 
response  rate  to  mail  questionnaires  tends  to  be  low,  the  secretaries  also  sent  two  mailings  of 
follow-up  reminders  regarding  the  questionnaire.  Questionnaires  were  numbered  so  that  if  one 
person  returned  more  than  one  questionnaire,  only  the  first  one  was  used  in  the  analyses  (i.e., 
at  all  three  mailouts,  person  1 would  receive  questionnaire  1,  and  so  on).  Pre-addressed, 
postage-paid  return  envelopes  were  provided  so  that  postage  was  paid  only  on  those 
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questionnaires  that  were  returned.  In  this  way,  all  responses  were  kept  anonymous  and  the 
student  had  no  access  to  the  names  of  clients  in  the  two  programs. 

Finally,  the  student  interviewed  the  executive  directors  of  the  two  out-of-county  detox  programs 
to  which  clients  of  the  assessment/referral  and  day  treatment  programs  had  been  referred. 
Specific  information  gathered  from  these  interviews  included  the  percentage  of  out-of-county 
clients  admitted,  the  number  of  counties  from  which  referrals  were  made,  and  the  procedures 
used  with  out-of-county  referrals  in  terms  of  intake  and  follow-up. 


d.  How  were  the  data  analyzed? 

Information  from  the  completed  questionnaires  was  coded  and  entered  in  the  computer  and 
analyzed  using  EPI  INFO  (see  Appendix  A).  Average  ratings  in  response  to  each  question  were 
calculated.  Additional  comments  were  summarized  in  the  report.  Information  from  treatment 
programs  about  individuals  who  did  not  use  existing  detox  services  despite  an  apparent  need,  and 
interview  data  from  the  directors  of  the  out-of-country  detox  programs  were  summarized  in  the 
report. 


e.  What  did  they  find  out? 

Two  hundred  and  ten  individuals  were  identified  by  the  assessment/referral  centre  or  the  day 
treatment  program  as  needing  detoxification  services  during  the  previous  two  years.  Of  these, 
165  (79%)  had  used  out-of-county  detox  services,  and  45  (21%)  received  no  detox  services. 

Of  the  45  clients  who  did  not  use  detox  services  despite  an  apparent  need,  18  (40%)  did  not  use 
the  outlying  detox  services  because  of  lack  of  transportation;  15  (33%)  were  denied  admission 
due  to  lack  of  space  within  the  detox  program;  nine  (20%)  were  denied  admission  because  they 
had  been  using  drugs  which  the  detox  did  not  feel  equipped  to  handle  (e.g.,  cocaine,  heroin, 
LSD);  and  three  (7%)  were  denied  admission  because  they  required  medical  attention  and  the 
services  were  strictly  non-medical. 

Of  the  165  individuals  who  were  sent  questionnaires  regarding  their  experiences  with  detoxes, 
100  (61%)  returned  completed  questionnaires.  The  average  ratings  for  all  potential  problems 
indicated  at  least  a moderate  problem  level.  Table  1 summarizes  the  average  ratings  of  problems 
potentially  experienced  by  clients  who  attended  an  out-of-county  detox  program. 
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Table  1.  Average  Ratings  of  Problems  Associated  with  Attending  an  Out-of-County 
Agency 


Average  problem  rating 
(1  - not  at  all  a problem; 

3 - very  much  a problem) 

lack  of  transportation 

2.4 

being  too  far  away  for  family  and  friends  to  visit 

2.5 

feeling  lonely  and  isolated 

2.6 

no  involvement  in  local  services  (e.g.,  AA,  NA) 

2.5 

no  information  about  further  treatment 

2.3 

no  plan  for  further  treatment 

2.1 

no  follow-up 

2.2 

The  two  out-of-county  detox  programs  had  similar  policies.  Both  detox  programs  allotted  only 
15%  of  their  beds  to  clients  from  outside  their  catchment  area  and  both  received  referrals  to 
these  beds  from  10  counties. 

For  those  detox  clients  who  attended  from  outside  the  programs’  catchment  areas,  formal 
arrangements  for  treatment  and  follow-up  services  in  the  clients’  home  county  were  rarely  made. 
Clients  from  within  the  detoxes’  catchment  area,  however,  were  often  assessed  while  in  detox 
and  admitted  directly  to  treatment  programs  upon  discharge.  As  well,  one  of  the  detox  programs 
included  follow-up  of  all  clients  from  its  own  catchment  area  on  a regular  basis  (one  and  three 
months  after  discharge)  to  assess  the  status  of  clients  and  to  find  out  whether  clients  had  attended 
treatment.  No  follow-up  of  clients  from  outside  their  main  catchment  area  was  done  by  either 
detox. 


/.  How  were  the  results  used? 

The  results  from  this  needs  assessment  provided  support  for  the  development  of  a local  detox. 
The  task  force  developed  a proposal  to  be  submitted  to  the  Ministry  of  Health  for  a local  detox. 
Since  development  of  the  detox  took  several  years,  steps  were  taken,  in  the  interim,  to  provide 
better  co-ordination  with  the  two  out-of-county  detox  programs.  The  program  administrators 
from  the  assessment/referral  and  day  treatment  programs  made  presentations  to  detox  staff  to 
inform  them  about  the  available  treatment  and  self-help  services  and  contacts  in  this  county.  The 
detox  programs  and  the  two  treatment  programs  formalized  a system  of  referral  from  detox 
directly  to  these  programs. 
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Methodological,  measurement  and  ethical  issues. 

Methodological.  For  a mailed-out  questionnaire,  the  return  rate  was  fairly  high  (61%). 
The  positive  response  may  have  been  due  to  the  follow-up  reminders,  the  short  easy  format 
of  the  questionnaire,  and  the  pre-addressed  postage-paid  envelopes  provided  with  the 
questionnaires.  All  of  these  are  factors  that  are  known  to  increase  response  rates.  While 
the  additional  follow-ups  add  to  the  cost  of  the  project,  maximizing  the  response  rate  is 
critical  in  obtaining  valid  information.  Even  a response  rate  of  61%  could  reflect  bias. 
For  example,  those  who  returned  questionnaires  may  have  been  more  likely  to  have  found 
using  out-of-county  detox  services  problematic;  while  those  who  experienced  fewer 
problems  may  have  tended  not  to  bother  returning  the  questionnaires.  Therefore,  the 
results  needed  to  be  interpreted  as  possibly  overestimating  the  demand  for  local  detox 
services. 

Ethical.  Since  the  staff  of  the  assessment/referral  and  day  treatment  programs  reviewed 
client  files  and  mailed  the  questionnaires,  it  was  possible  to  maintain  client  confidentiality. 
This  method  ensured  that  no  one  outside  the  two  treatment  programs  had  access  to  client 
names. 
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Follow-Up  on  Users  of  Out-of-County  Detox  Programs 


Conducted  by  the Assessment  and  Referral  Program  (or  the 

Day  Treatment  Program)  on  behalf  of  the County  Substance  Abuse  Task  Force. 


Please  rate  the  degree  to  which  you  experienced  the  following  problems  while  attending 
detox  outside  this  county: 


Not  at 

Somewhat 

Very 

all  a 

of  a 

much  a 

problon 

problem 

problem 

Lack  of  transportation 

1 

2 

3 

Being  too  far  away  for  family  and  friends  to  visit 

1 

2 

3 

Feeling  lonely  and  isolated 

1 

2 

3 

No  involvement  with  local  services  (e.g.,  AA,  NA) 

1 

2 

3 

No  information  about  further  treatment 

1 

2 

3 

No  plan  for  further  treatment 

1 

2 

3 

No  follow-up 

1 

2 

3 

Do  you  have  any  additional  comments  regarding  your  experience  using  detox  services? 


Thank  you! 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  moderate 


Part  2.  Client  Characteristics 


Did  clients  who  reported  experiencing  sexual  or  physical  abuse  enter  the 
detox  program  more  often  than  clients  who  reportM  no  abuse? 


a.  Who  was  asking  the  question  and  why  did  they  want  this  information? 

People  admitted  to  detox  were  asked  routinely  whether  they  had  ever  experienced  sexual  or 
physical  abuse.  Where  possible,  detox  staff  attempted  to  refer  those  who  had  experienced  abuse 
to  counselling  or  a support  group  in  the  community.  Some  clients  did  not  want  a referral.  For 
those  who  did,  the  kind  of  help  they  felt  they  needed  was  often  not  available. 

Intake  statistics  indicated  that  a high  proportion  of  detox  clients  (48%)  reported  having  been 
sexually  or  physically  abused.  There  were  limited  resources  available  in  the  community, 
however,  for  counselling  people  with  abuse  histories,  particularly  those  who  were  also 
experiencing  problems  with  alcohol  and  drug  abuse.  The  director  and  staff  had  observed  that 
clients  with  abuse  histories  seemed  to  be  admitted  more  frequently  to  the  detox.  They 
hypothesized  that  one  reason  for  more  frequent  admissions  was  that  abused  women  and  men 
were  not  receiving  the  help  they  needed  elsewhere,  and  continued  to  use  alcohol  and  other  drugs 
to  cope  with  abuse  issues. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

The  detox  program  had  implemented  a customized  computer  database  program  that  allowed  them 
to  record  client  intake  information,  as  well  as  tracking  client  readmissions.  The  program  staff 
member  who  maintained  the  database  was  asked  to  conduct  the  analyses  required  to  assess 
whether  clients  who  had  abuse  histories  had  been  readmitted  more  frequently  during  the  previous 
year.  The  data  analyses  (using  the  customized  program)  and  preparation  of  summary  graphs 
(using  Harvard  Graphics  [see  Appendix  C])  required  approximately  two  hours. 
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c.  How  were  the  data  collected? 

Data  were  collected  as  part  of  the  detox  operation. 


d.  How  were  the  data  analyzed? 

The  detox  used  a customized  computer  program  that  automatically  calculated  the  number  of 
admissions  per  year.  To  address  the  current  question,  the  average  number  of  admissions  was 
calculated  separately  for  those  who  reported  abuse  and  for  those  who  did  not  report  abuse.  The 
analyses  were  also  calculated  separately  for  males  and  females. 

Harvard  Graphics  was  used  to  produce  bar  graphs. 


e.  What  did  they  find  out? 

Of  the  480  clients  who  had  used  the  detox  during  the  previous  year,  228  (48%)  reported  that 
they  had  been  sexually  or  physically  abused  at  least  once  in  the  past.  Among  female  clients, 
78%  (47  out  of  60)  reported  a history  of  abuse;  for  men,  the  rate  was  lower,  with  43%  (181  out 
of  420)  reporting  a history  of  abuse  (see  Figure  1,  produced  using  Harvard  Graphics). 

Those  who  reported  a history  of  abuse  had  been  admitted  to  the  detox  an  average  of  3.1  times 
during  the  past  year;  whereas  clients  who  reported  no  abuse  had  been  admitted  an  average  of 
1.6  times  (see  Figure  2).  Male  clients  had  been  admitted  to  detox  an  average  of  2.3  times 
during  the  past  year,  and  female  clients  an  average  of  2.2  times  (see  Figure  2).  Separate 
analyses  for  males  and  females  indicated  that  a history  of  abuse  increased  the  average  number 
of  admissions  for  both  males  and  females.  Figure  3 illustrates  in  bar  graph  form  the  average 
number  of  admissions  of  male  and  female  clients  who  reported  a history  of  abuse  compared  to 
those  who  did  not. 
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Figure  1 . Percent  of  Male  and  Female  Detox  Clients  Reporting  a 

History  of  Abuse 

0%  20%  40%  60%  80%  100% 

Female 

Male 


Figure  2.  Average  Number  of  Admissions  to  Detox  in  One  Year  by 
History  of  Abuse  and  Gender  of  Clients 
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Figure  3.  Average  Number  of  Admissions  to  Detox  in  One  Year  for 
Ciients  with  and  Without  a History  of  Abuse  (Male  Compared  to 

Female  Clients) 

0 0.5  1 1.5  2 2.5  3 3.5  4 


History  of  abuse 


No  history  of  abuse 


■ Mala  (ns420)  ^Female  (ns60) 


/.  How  were  the  results  used? 

The  data  provided  confirmation  of  the  staffs  impression  that  clients  who  had  been  abused  were 
admitted  to  the  detox  more  often.  The  staff  and  director  felt  they  had  sufficient  evidence  to 
devote  some  time  to  improving  services  for  clients  who  had  abuse  histories  and  substance  abuse 
problems.  The  director  committed  time  to  seeking  out  additional  community  services  willing 
and  qualified  to  counsel  these  clients.  The  director  also  intended  to  use  this  time  to  raise 
awareness  in  the  community  (among  both  addictions  treatment  services  and  services  for  battered 
or  sexually  abused  men  and  women)  about  the  link  between  abuse  and  alcohol  and  drug 
problems.  To  further  these  ends,  the  director  made  presentations  to  the  DHC,  addictions 
programs,  women’s  programs,  and  other  community  groups  and  was  able  to  obtain  some  media 
exposure. 

The  results  of  these  efforts  included  the  following:  the  identification  of  five  additional  services 
prepared  to  accept  male  and  female  detox  referrals  for  counselling  regarding  sexual  and  physical 
abuse;  the  development  of  a one-day  workshop  where  staff  of  women’s  programs  and  addictions 
treatment  programs  met  to  share  their  expertise  and  develop  better  linkages  for  women  needing 
both  types  of  services. 
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Level  of  difficulty:  difficult 
Level  of  resources  needed:  heavy 


Part  3.  Program  Delivery 


Did  pre-treatment  counselling  (including  a longer  stdyy  ^ d^^  increase 
the  proportion  of  clients  who  accepted  treatment  f^frals  and  reduce  the 
average  rate  of  readnUssions? 


a.  Who  was  asking  the  question  and  why  did  they  want  this  information? 

The  board  requested  that  the  director  and  staff  investigate  pre-treatment  counselling  as  a way 
to  increase  referrals  to  addictions  treatment.  While  part  of  the  detox’s  function  is  to  provide  a 
safe  haven  for  the  inebriated  person,  the  detox  is  also  supposed  to  function  as  a first  step  in 
linking  people  to  treatment.  The  program  staff  had  provided  referrals  to  treatment  where 
possible,  but  had  little  time  to  counsel  clients  and  facilitate  entry  to  treatment.  The  detox  was 
able  to  secure  one-year  funding  for  an  extra  person  to  provide  pre-treatment  counselling  (both 
individual  and  group  counselling,  as  appropriate)  for  clients  at  the  detox.  This  person  would 
also  identify  treatment  needs  and  encourage  referrals  to  treatment.  Although  this  counselling 
was  likely  to  increase  the  length  of  stay  in  detox,  it  was  hoped  that  the  longer  length  of  stay 
would  be  counteracted  by  more  clients  entering  treatment  (increasing  the  likelihood  that  they 
would  resolve  substance  abuse  problems)  and  fewer  clients  being  readmitted. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

First,  the  project  involved  an  extra  full-time  person  who  provided  pre-treatment  counselling. 
This  person  also  trained  other  staff  to  do  pre-treatment  counselling  and  conducted  follow-up  with 
programs  to  which  clients  were  referred  to  find  out  whether  clients  entered  and  completed  the 
program.  In  addition,  at  the  end  of  the  year  during  which  pre-treatment  counselling  was 
provided,  the  program  secretary  spent  six  hours  compiling  statistics  and  preparing  graphs 
regarding  the  proportion  of  clients  referred  for  addictions  treatment,  average  length  of  stay,  and 
number  of  readmissions  during  the  three  years  preceding  the  pre-treatment  year  compared  to 
during  the  pre-treatment  year.  The  detox  already  used  a customized  computer  database  that 
contained  all  the  necessary  information. 
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c.  How  were  the  data  collected? 

Statistics  regarding  referrals  and  readmissions  were  kept  as  a routine  part  of  the  detox  operation. 
During  the  year  of  pre-treatment  counselling,  the  counsellor  followed  up  all  referrals  and  kept 
records  of  referrals  accepted  by  clients  (including  whether  they  entered  and  whether  they 
completed  the  program  to  which  they  had  been  referred). 


d.  How  were  the  data  analyzed? 

The  secretary  generated  the  following  statistics  from  a customized  computer  database: 
proportion  of  clients  referred  for  addictions  treatment,  average  length  of  stay  (in  days)  at  detox, 
and  average  number  of  readmissions  per  client.  The  counsellor  calculated  the  following  by 
hand:  the  proportion  of  referrals  where  the  client  entered  the  program  to  which  he  or  she  had 
been  referred  and  the  proportion  of  referrals  where  the  client  received  a satisfactory  discharge 
from  the  program.  While  statistics  were  kept  on  all  referrals  made,  only  referrals  to  addictions 
treatment  programs  are  included  in  the  present  analyses. 

Area  graphs  showing  the  results  were  generated  using  Harvard  Graphics  (see  Appendix  C). 


e.  What  did  they  find  out? 

Statistics  for  the  three  years  prior  to  the  pre-treatment  counselling  program  and  for  the  year  of 
the  program  are  reported  in  Table  1 . The  same  results  are  shown  in  graphic  form  using  Harvard 
Graphics  in  Figures  1,  2 and  3. 

These  results  indicated  that  the  program  had  achieved  some  success.  More  clients  were  being 
referred  to  treatment,  and  although  the  length  of  stay  had  increased  slightly,  the  number  of  re- 
admissions had  decreased  by  over  a half. 
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Table  1.  Proportion  of  Clients  Referred  for  Addictions  Treatment,  Average  Length  of 
Stay,  Average  Number  of  Readmissions 


Year! 

Year  2 

Year  3 

Year  of  pre-treatment 
counselling  program 

proportion  of  clients  referred 
for  addictions  treatment  (all 
admissions) 

20% 

24% 

19% 

39% 

average  length  of  stay  (all 
admissions) 

3.2  days 

2.9  days 

2.9  days 

4.4  days 

average  number  of  readmissions 
(excluding  first  admission) 

1.3 

1.6 

1.4 

.6 

Figure  1 . Percent  of  Detox  Clients  Referred  for  Addictions 
Treatment  (Year  of  Pre-Treatment  Counselling  Compared  With  the 
First  Three  Years  of  Operation  of  the  Detox) 


Year  1 


Year  2 


Year  3 Year  of  pre-treatment 
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Rgure  2.  Average  Length  of  Stay  at  Detox  (in  Days) 
(Year  of  Pre-Treatment  Counselling  Compared  With 
the  First  Three  Years  of  Operation  of  the  Detox) 
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Year  1 Year  2 Year  3 Year  of  pre-treatment 


Figure  3.  Average  Number  of  Readmissions  to  Detox 
(Year  of  Pre-Treatment  Counselling  Compared  With 
the  First  Three  Years  of  Operation  of  the  Detox) 
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According  to  the  information  collected  in  follow-up  interviews  with  clients  who  had  been 
referred  for  addictions  treatment,  85%  said  they  followed  through  on  the  referral  and  78% 
reported  receiving  a satisfactory  discharge  from  the  program.  In  this  project,  the  counsellor 
completed  follow-ups  only  with  clients  (not  with  the  agencies  to  which  clients  had  been 
referred).  Example  F describes  issues  regarding  follow-up  with  both  clients  and  agencies. 
Although  no  statistics  were  available  regarding  follow-through  on  referrals  from  previous  years, 
results  suggested  a high  rate  of  successful  referrals. 


/.  How  were  the  results  used? 

The  results  confirmed  the  usefulness  of  pre-treatment  counselling.  Because  readmissions  were 
down,  it  was  felt  that  the  staff  would  now  be  able  to  direct  more  time  to  pre-treatment 
counselling.  Although  this  could  not  be  done  as  easily  as  when  a full-time  person  was  available 
for  the  task,  the  program  director  felt  that  they  could  do  a reasonably  good  job  of  it  now  that 
detox  staff  were  trained  and  had  a little  more  time  for  the  counselling.  However,  they  would 
continue  to  monitor  the  effectiveness  of  the  pre-treatment  counselling  program  in  terms  of 
referral  and  re-admission  rates. 


Methodological,  measurement  and  ethical  issues: 

Methodological.  There  are  annual  variations  in  referrals  and  readmissions  that  occur  by 
chance.  Therefore,  it  was  important  to  compare  pre-treatment  effects  of  several  years,  not 
just  the  year  before.  While  the  effect  of  pre-treatment  counselling  appeared  large,  it  is 
important  to  continue  monitoring  the  effect  to  ensure  that  the  results  are  stable  and  not  just 
attributable  to  chance  or  to  the  specific  counsellor  who  was  hired  for  the  one-year  period. 
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Program:  The  program  is  a substance  abuse  assessment/referral  service.  The  mandate  of  the 
program  is  to  perform  a comprehensive  assessment  on  every  client  and  to  refer  clients  to 
appropriate  services.  Services  offered  by  the  program  are  intake,  assessment,  referral,  case 
management,  brief  counselling,  and  community  education  and  development. 

Initial  contact  can  be  made  with  the  program  by  other  agencies  on  a person’s  behalf  (formal 
referral)  or  by  a person  directly  contacting  the  program  (self-referral).  A two-hour  assessment 
interview  is  scheduled  at  this  point.  There  is  generally  a two  to  three  month  waiting  list  of 
people  to  be  assessed.  However,  there  are  sometimes  exceptions  to  this  and  clients  may  be 
given  an  appointment  sooner.  Once  assessed,  clients  may  be  referred  for  case  management, 
substance  abuse  treatment,  or  other  services.  Some  require  no  further  service  after  the 
assessment. 

This  example  includes  quality  enhancement  questions  for  four  levels  of  evaluation: 

Part  1.  Logic  Model 

Part  2.  Program  Delivery  I 

Part  3.  Program  Delivery  n 

Part  4.  Consumer  Satisfaction  - Agency  Survey. 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  moderate 

Part  2.  Program  Delivery  I 

What  was  the  average  number  of  days  hetweenintake  and  firsts^ 
assessment  appointment? 

What  proportion  of  referrals  attended  the  origthally  scheduled  assessment 
interview? 

What  proportion  of  referrals  were  assessed  on  a subsequent  appointment? 
Overall,  what  proportion  of  assessment  appointments  resulted  mho-shows? 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 

The  program  manager  and  staff  had  been  concerned  that  many  of  those  who  scheduled  an 
assessment  did  not  keep  the  appointment.  While  all  no-shows  caused  inconvenience,  no-shows 
for  assessment  appointments  resulted  in  considerable  inefficiency.  Each  initial  assessment 
appointment  scheduled  was  allotted  3.5  hours  of  counsellor  time  for  intake,  assessment 
interview,  liaison  with  referral  source  and  other  involved  parties,  report  writing,  and  treatment 
planning.  Although  the  time  was  used  for  other  purposes  when  a client  did  not  attend  the 
appointment,  the  amount  of  time  that  needed  to  be  allotted  to  initicil  assessment  limited  the 
number  of  assessment  appointments  that  could  be  scheduled  for  any  particular  day,  and 
consequently  limited  the  number  of  new  admissions  to  the  program. 

The  program  manager  had  decided  to  implement  group  orientation  sessions  to  try  to  reduce  drop- 
out between  intake  and  assessment  and  to  reduce  the  number  of  assessment  appointments  where 
the  client  does  not  show  up.  Before  undertaking  this  program  change,  basic  statistics  on  no- 
shows  for  initial  assessment  appointments  were  calculated  for  a one-year  period. 
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b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

In  order  to  keep  statistics  on  appointments  and  attendance,  a computer  and  customized  software 
were  needed.  The  software  was  designed  to  extract  program  information  such  as  days  lapsed 
between  date  of  intake  and  date  of  assessment  appointment,  attendance  at  appointments,  and  so 
on. 


The  program  secretary  maintained  the  database  on  appointments  and  other  services  and  updated 
it  daily.  It  took  the  secretary  several  hours  to  extract  the  statistics  required  to  address  these 
questions  (based  on  data  entered  into  the  computer  during  the  previous  year)  and  to  prepare 
graphs  using  Harvard  Graphics. 


c.  How  were  the  data  collected? 

Attendance  data  were  recorded  on  daily  appointment  logs  kept  by  the  assessment  workers.  The 
intake  worker  kept  a record  of  the  date  of  the  initial  intake  and  the  date  of  the  initially  scheduled 
assessment  appointment.  If  a change  was  made  in  the  date  of  the  initial  appointment,  the  intake 
worker  reported  this  to  the  secretary  who  updated  the  information  in  the  database. 


d.  How  were  the  data  analyzed? 

The  information  required  for  this  evaluation  already  existed  in  the  database.  The  program 
secretary  computed  descriptive  statistics  for  the  following: 

(a)  average  number  of  days  between  intake  and  first  scheduled  assessment  appointment 

(b)  proportion  of  referrals  who  attended  the  originally  scheduled  assessment  appointment 

(c)  proportion  of  referrals  assessed  at  a later  appointment 

(d)  overall  proportion  of  assessment  appointments  where  the  client  did  not  show. 

Harvard  Graphics  (see  Appendix  C)  was  used  to  generate  a graphical  presentation  of  the  results 
in  pie  graph  form. 


e.  What  did  they  find  out? 

There  were  984  clients  who  scheduled  at  least  one  assessment  interview  during  the  one-year 
period.  The  average  length  of  time  between  intake  and  the  first  assessment  appointment  was  67 
days.  Of  the  984  clients  who  scheduled  assessment  interviews,  44%  attended  the  originally 
scheduled  appointment,  17%  attended  a later  appointment,  and  39%  scheduled  one  or  more 
appointments  but  never  attended  (see  Table  1).  Figure  1 shows  these  proportions  as  a pie  chart 
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using  Harvard  Graphics  (see  Appendix  C).  Including  rescheduled  assessment  appointments, 
1,725  assessment  interviews  were  scheduled  for  984  clients  during  the  one-year  period. 


Table  1.  Proportion  of  Clients  Who  Attended  the  Original  Assessment,  a Later 
Assessment,  or  Did  Not  Attend  Assessment 


Number  and  % of  clients 

clients  who  attended  originally  scheduled  assessments 

430 

(44%) 

clients  who  attended  assessment  at  a later  appointment 

168 

(17%) 

clients  who  scheduled  one  or  more  assessment  interviews, 
but  never  attended 

386 

(39%) 

Figure  1 . Proportion  of  Clients  Who  Scheduled  at  Least  One 
Assessment  interview  and  Attended  the  Originally  Scheduled 
Appointment,  a Later  Appointment,  or  Did  Not  Attend  an  Appointment 


Attended  originally 
scheduled  appointment 

(430)  44% 


Attended  later 
appointment 

(168)  17% 


did  not  attend 


(386)  39% 
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The  results  indicated  that  the  current  intake  component  needed  to  be  improved  in  order  to  meet 
the  short-term  objective  of  preventing  drop-out/no-shows.  The  program  manager  decided  to 
proceed  with  adding  group  orientation  to  the  intake  phase  of  the  program.  Since  decreasing  the 
number  of  clients  who  did  not  show  up  for  initial  assessment  would  increase  the  number  of  new 
clients  who  could  be  assessed,  adding  the  orientation  component  would  be  expected  to  decrease 
the  waiting  list.  As  well,  clients  who  completed  assessment  would  be  more  likely  to  begin  the 
treatment  process  and  more  likely  to  achieve  long-term  recovery  from  substance  abuse. 
Therefore,  increasing  the  proportion  of  clients  who  completed  assessment  would  be  expected  to 
improve  the  ultimate  outcomes  of  clients. 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  moderate 


Part  3.  Program  Delivery  n 


To  what  extent  did  requiring  attendance  at  two  group  orientation  sessions 
before  scheduling  the  assessment  interview  reduce  the  rate  of  dropout 
between  intake  and  assessment? 

To  what  extent  did  the  requirement  to  attend  group  orientation  reduce  the 
rate  of  no-shows  at  the  originally-schedtded  assessment? 

To  what  extent  did  adding  group  orientation  to  the  intake  phase  of  the 
program  result  in  a reduction  in  the  waiting  list  period  betWeeti  intake  and 
assessment? 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 

It  is  the  short-term  objective  of  the  intake  component  of  the  addiction  assessment/referral  centre 
to  prevent  no-shows/drop-outs  at  the  assessment.  In  Part  2 of  this  example,  baseline  statistics 
indicated  that  almost  40%  of  referrals  dropped  out  between  intake  and  assessment  (i.e., 
scheduled  but  never  attended  assessment).  It  was  decided  by  the  managers  and  the  board  to  add 
two  group  orientation  sessions  to  the  intake  component  of  the  agency.  The  objective  of  these 
sessions  was  to  introduce  clients  to  the  services  offered  by  the  agency  and  to  provide  some 
immediate  response  to  clients  who  were  having  to  wait  a long  time  for  an  assessment 
appointment. 

Since  attendance  at  the  sessions  would  allow  personal  contact  between  clients  and  agency  staff 
within  one  month  following  the  initial  contact,  it  was  hoped  that  this  personal  contact  would 
reduce  the  number  of  people  who  dropped  out  between  intake  and  assessment  and  increase  the 
rate  of  attendance  at  the  initially-scheduled  assessment  appointment.  The  process  would  also  be 
more  efficient  because  those  who  were  really  not  interested  in  proceeding  with  an  assessment 
would  realize  this  during  orientation  and  not  schedule  an  assessment  appointment.  Fewer  no- 
shows  at  assessment  and  fewer  unwanted  assessments  scheduled  should  also  reduce  the  waiting 
period  between  intake  and  assessment.  Thus,  the  intention  of  the  group  orientations  was  to  (a) 
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reduce  the  percentage  of  people  who  dropped  out  between  intake  and  assessment,  (b)  decrease 
the  number  of  no-shows  at  assessment,  cmd  (c)  reduce  the  waiting  period. 

Once  the  orientation  sessions  had  been  in  place  for  a year,  the  program  manager  had  new 
statistics  calculated  regarding  the  rate  of  no-shows  and  the  average  waiting  period. 


h.  What  resources  were  needed  to  collect  and  interpret  the  information? 

The  program  secretary  maintained  and  updated  daily  a database  on  appointments  and  other 
services  (see  Part  2 of  this  example  for  a description  of  the  database).  It  took  the  secretary  four 
hours  to  calculate  the  required  statistics  and  prepare  tables  and  graphs  of  the  results. 


c.  How  were  the  data  collected? 

The  date  of  initial  intake  was  recorded  by  the  intake  worker.  Attendance  at  orientation  sessions 
was  recorded  by  the  session  counsellors.  The  date  of  the  initially  scheduled  assessment 
appointment  was  recorded  by  the  intake  worker  after  the  second  orientation  session.  The 
attendance  at  the  assessment  appointment  was  recorded  on  daily  appointment  logs  kept  by  the 
assessment  workers.  If  a change  was  made  in  the  date  of  the  initial  appointment,  the  intake 
worker  reported  this  to  the  secretary  who  updated  the  information  in  the  database. 


d.  How  were  the  data  analyzed? 

The  information  for  this  evaluation  already  existed  in  the  database.  The  program  secretary 
computed  descriptive  statistics  for  the  following: 

(a)  proportion  of  referrals  who  attended  at  least  one  orientation  session 

(b)  proportion  of  referrals  who  scheduled  an  assessment  appointment 

(c)  proportion  of  referrals  who  attended  the  originally  scheduled  assessment  appointment 

(d)  proportion  of  referrals  who  were  assessed  at  a later  appointment 

(e)  proportion  of  referrals  who  never  attended  an  assessment  appointment 

(f)  average  number  of  days  between  intake  and  first  scheduled  assessment  appointment. 

These  statistics  were  then  compared  with  those  for  the  year  before  the  orientation  sessions  were 
implemented. 

Graphical  presentation  of  the  results  were  generated  using  Harvard  Graphics  (see  Appendix  C). 
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e. 


There  were  1,062  referrals  during  the  12-month  period  following  the  introduction  of  orientation 
sessions.  Table  1 summarizes  the  proportion  of  clients  attending  orientation  sessions,  scheduling 
assessment  appointments,  and  attending  assessment  appointments,  comparing  the  year  with 
orientation  sessions  to  the  previous  year.  Figure  1 shows  the  same  result  in  pie  chart  form 
produced  using  Harvard  Graphics. 


Table  1.  Attendance  at  Scheduled  Appointments  Before  and  After  the  Implementation 
of  Orientation  Sessions 


Before  oricntdtibii 
sessions  were  part  of 
intake 

After  orientations 
becartie  part  of 
intake 

number  of  clients  accepted  to  the  program 
through  intake 

984 

1,062 

percent  of  clients  attending  at  least  one 
orientation 

— 

987  (93%) 

percent  of  clients  who  scheduled  at  least  one 
assessment  appointment 

984  (100%) 

898  (91%) 

percent  of  clients  who  scheduled  assessment, 
and  attended  originally-scheduled  appointment 

430  (44%) 

657  (73%) 

percent  of  clients  who  attended  assessment  at  a 
later  appointment 

168  (17%) 

92  (10%) 

percent  of  clients  who  scheduled  assessment  but 
never  attended 

386  (39%) 

149  (17%) 

percent  of  all  clients  who  were  accepted  to  the 
program  who  never  attended  an  assessment 

386  (39%) 

313  (29%) 

To  assess  the  effects  of  the  orientation  program  on  the  waiting  period,  the  length  of  the  waiting 
period  was  averaged  for  each  quarter  of  the  year  following  the  introduction  of  the  orientation 
sessions.  During  the  year,  the  waiting  period  dropped  from  64  days  in  the  first  quarter,  to  54 
days,  39  days  and,  fincilly,  30  days  in  the  final  quarter.  These  results  are  shown  in  an  area 
graph  produced  using  Harvard  Graphics  in  Figure  2. 
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Figure  1 . Proportion  of  Assessment  Appointments  Scheduled  That 
Were  Attended  (The  Year  Before  the  Orientation  Sessions  Were 
Implemented  Compared  to  the  Year  After) 
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Assessed  as  scheduled 


Assessed  eventually 
(168)  17% 


Before 


Never  assessed 
(149)  17% 


Never  assessed 
(386)  39% 


Assessed  eventually 
92)  10% 


Assessed  as  scheduled 
(657)  73% 

After 


Rgure  2.  Average  Quarterly  Waiting  Period  for  Assessment  During 
the  Year  Following  Implementation  of  the  Orientation  Sessions 


1st  Quarter  2rK]  Quarter  3rd  Quarter  4th  Quarter 
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The  results  indicated  that  the  proportion  of  referrals  to  the  program  who  were  never  assessed 
(i.e.,  those  who  dropped  out  between  intake  and  assessment)  has  decreased  from  39%  prior  to 
the  implementation  of  the  orientation  sessions  to  29%  following  the  requirement  that  potential 
clients  attend  orientation  sessions.  Thus,  more  people  overall  were  entering  the  treatment 
process  through  attending  assessment. 

The  results  also  indicated  that  the  proportion  of  clients  assessed  as  originally  scheduled 
increased  from  44%  to  73%  following  the  introduction  of  the  orientation  sessions.  As  well,  the 
proportion  of  all  assessment  appointments  scheduled  which  were  not  attended  decreased  from 
65%  to  39%. 

Finally,  the  results  indicated  that  the  length  of  the  waiting  period  decreased  as  the  year 
progressed.  During  the  initial  two  quarters  of  the  year,  the  waiting  period  remained  long 
because  individuals  who  had  scheduled  appointments  before  implementation  of  the  orientation 
sessions  were  still  waiting  to  be  seen.  By  the  middle  of  the  year,  the  length  of  the  waiting 
period  had  decreased  substantially. 

These  findings  suggested  that  the  implementation  of  the  orientation  sessions  helped  the  program 
to  meet  its  short-term  objective  of  preventing  no-shows/drop-outs  at  the  assessment. 
Correspondingly,  the  length  of  the  waiting  period  decreased  substantially  because  there  were 
fewer  appointments  being  rescheduled.  Therefore,  the  program  manager  and  staff  decided  to 
continue  to  require  attendance  at  the  orientation  sessions  before  scheduling  an  assessment 
appointment.  They  plan  to  continue  to  monitor,  on  an  annual  basis,  the  drop-out  rates  between 
intake  and  assessment,  the  attendance  rates  at  assessment  appointments,  and  the  length  of  the 
waiting  period. 


Methodological,  measurement  and  ethical  issues: 

Methodological.  The  waiting  period  depends  both  on  the  efficient  use  of  program  staff  as 
well  as  the  number  of  referrals  made  to  the  program.  In  this  example,  the  total  number  of 
persons  referred  to  the  program  stayed  about  the  same.  If,  for  some  reason,  the  number  of 
referrals  to  the  program  had  gone  up  or  down  substantially,  this  would  have  affected  the 
waiting  period  and  would  have  made  a comparison  of  the  waiting  period  for  the  two  years 
meaningless  in  terms  of  assessing  the  effects  of  the  orientation  session. 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  moderate 


Part  4.  Consumer  Satisfaction  - Agency  Survey 


Were  community  agencies  aware  of  the  services  offered  by  the 
assessment/referral  program? 

To  what  extent  have  community  agencies  referred  clients  to  the 
assessment/referral  program  in  the  past? 

How  satined  were  community  agencies  with  the  services  of^^  the 
assessment/referral  program  ? 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 

The  program  manager  and  staff  wanted  to  ensure  that  community  agencies  were  aware  of  the 
types  of  services  they  provided  and  that  they  had  been  responding  satisfactorily  to  referrals  made 
to  the  program  in  the  past.  A brief  survey  was  developed  to  address  these  issues. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

The  program  manager  and  staff  spent  three  hours  devising  a questionnaire  to  be  sent  to  the 
community  agencies  who  might  refer  to  the  service.  The  secretary  spent  six  hours  preparing  the 
questionnaires  and  mailing  them  to  the  community  agencies.  The  secretary  then  spent  six 
additional  hours  tallying  information  from  the  completed  questionnaires  and  preparing  tables  and 
graphs. 
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c.  How  were  the  data  collected? 

A mailing  list  of  community  agencies  had  previously  been  prepared  as  the  assessment/referral 
service  had  identified  relevant  agencies  when  they  had  promoted  a workshop  a few  months 
earlier.  The  program  secretary  mailed  the  questionnaire  to  the  managers  of  38  community 
agencies.  Completed  questionnaires  were  returned  to  the  program  secretary  who  then  tallied  the 
information  by  hand. 

Because  the  manager  was  interested  in  how  the  program  had  responded  to  particular  agencies, 
as  well  as  overall  statistics,  questionnaires  were  not  anonymous.  To  ensure  a high  return  rate, 
the  program  secretary  made  a reminder  telephone  call  to  programs  who  had  not  returned  the 
questionnaire  by  three  weeks  after  the  initial  mailing. 


d.  How  were  the  data  analyzed? 

Frequencies  and  average  ratings  were  calculated  and  general  comments  were  summarized. 
Bar  graphs  presenting  the  results  were  generated  using  Harvard  Graphics  (see  Appendix  C). 


e.  What  did  they  find  out? 

Of  38  questionnaires  that  were  mailed  out,  35  (92.1%)  were  returned.  All  respondents  were 
aware  of  the  assessment  and  planning  function  of  the  program,  but  fewer  were  aware  of  the  case 
management  component  and  very  few  (9%)  knew  about  the  brief  counselling  aspect  of  the 
program.  Two-thirds  of  the  respondents  reported  that  they  had  referred  clients  to  the  program 
and  most  indicated  high  satisfaction  with  all  aspects  of  the  program  except  the  waiting  period. 
Most  respondents  reported  that  they  intended  to  refer  clients  to  the  program  in  the  future.  These 
results  are  summarized  in  Table  1 . Awareness  of  program  services  and  satisfaction  ratings  are 
shown  in  bar  charts  in  Figures  1 and  2,  prepared  using  Harvard  Graphics. 

Comments  received  from  those  who  had  not  used  the  assessment/referral  services  in  the  past 
indicated  that  the  length  of  the  waiting  period  was  the  main  reason  that  they  did  not  refer  clients 
to  the  assessment/referral  centre.  Other  reasons  for  not  referring  to  the  assessment/referral 
program  included:  preferred  to  conduct  in-house  assessment  of  clients;  had  not  had  the  need  to 
refer  clients;  had  generally  referred  clients  to  their  family  physicians  for  substance  abuse  issues; 
agencies  were  not  aware  of  the  additional  services  offered  and  did  not  feel  it  would  be 
appropriate  to  refer  someone  for  assessment  if  no  further  services  would  be  offered. 
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Table  1.  Results  of  Questionnaires  Sent  to  Conununity  Agencies  (n=35) 


Questionnaire  item: 

Number  of  agencies  aware  of  the  following  services 
offered: 

assessment 

35 

(100%) 

treatment  planning 

35 

(100%) 

case  management 

26 

(74%) 

brief  counselling 

3 

(9%) 

Number  of  agencies  who  have  referred  clients  to  the 

24 

assessment/referral  centre 

(68.6%) 

Average  ratings  of  satisfaction  with  the  services  offered 
(n=24): 

4.8 

confidentiality 

2.1 

waiting  period 

4.1 

accessibility 

4.9 

ease  of  referral 

4.0 

helpfulness  of  staff 

Number  of  agencies  who  plan  to  refer  clients  to  the 

30 

assessment/referral  centre  in  the  future 

(85.7%) 
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Figure  1 . Percent  of  Community  Agencies  Aware  of  Program 
Services  Offered  by  the  Assessment/Referral  Program 
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Program  Services  o%  20%  40%  eo%  80%  100%  120% 


Assessment 


Treatment  Planning 

Case  Management 
Brief  Counselling 


Figure  2.  Average  Ratings  of  Satisfaction  with  Services  Offered  by 
the  Assessment/Referral  Program  by  Agencies  who  had  Made 
Referrals  to  the  Program  in  the  Past 

Program  Services  0123456 
Confidentiality 

Waiting  period 

Accessibility 

Ease  of  referral 

Helpfulness  of  staff 
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/.  How  were  the  results  used? 

First,  the  questionnaire  generally  helped  to  make  local  agencies  aware  of  the  services  offered 
by  the  assessment/referral  program. 

Second,  the  results  indicated  that  community  agencies  who  had  used  the  services  had  been 
generally  satisfied  with  the  assessment/referral  centre,  except  for  the  lengthy  waiting  period. 
Since  the  waiting  period  had  been  reduced  through  the  implementation  of  mandatory  attendance 
at  orientation  sessions,  the  program  manager  decided  to  write  a letter  to  each  community  agency, 
telling  them  the  results  of  the  survey  and  informing  them  that  recent  changes  in  procedures  had 
resulted  in  cutting  the  waiting  period  in  half.  The  letter  would  also  explain  that  particular  cases 
could  be  seen  even  sooner,  if  requested. 

Finally,  the  results  of  the  survey  were  made  available  to  staff,  the  board  and  other  interested 
persons.  The  positive  feedback  from  community  agencies  helped  boost  staff  morale. 


Methodological,  measurement  and  ethical  issues: 

Methodological.  One  of  the  problems  in  evaluating  referral  practices  is  that  different 
agencies  define  "referral"  in  different  ways.  For  example,  agency  staff  may  suggest  that 
clients  go  to  a program,  but  it  is  only  an  official  "referral"  when  a referral  form  is  sent  to 
the  referred-to  agency.  The  assessment/referral  program  was  interested  in  referrals  made 
to  them  using  the  broadest  definition  of  referral.  Therefore,  the  questionnaire  asked  "Have 
you  ever  directed  anyone  to  our  agency?" 
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Name  of  agency 

Agency  Feedback  Survey 


Please  take  a few  minutes  to  give  us  your  opinion  of  our  agency.  Your  feedback  will  help  us 
improve  our  services. 


1.  We  are  interested  in  Hnding  out  whether  people  know  about  all  the  services  we  provide.  The 
following  is  a list  of  our  services.  Are  you  aware  of  any  of  our  services? 

(Please  check  (\f ) those  you  are  aware  of. ) 


Assessment 

□ 

Treatment  planning 

□ 

Case  management 

□ 

Brief  counselling 

□ 

2.  Have  you  ever  directed  anyone  to  our  agency? 

Yes  □ 

No  □ 


If  no,  are  there  particular  reasons  why  you  have  not  used  our  program? 


3.  If  you  have  had  experience  in  directing  clients  to  our  agency,  how  satisfied  have  you  been  with 


the  following? 

Not  at  all 
satisfied 

Very 

satisfied 

Don’t  know 

Confidentiality 

1 

2 

3 

4 

5 

Waiting  period 

1 

2 

3 

4 

5 

Accessibility 

1 

2 

3 

4 

5 

Ease  of  referral 

1 

2 

3 

4 

5 

Helpfulness  of  staff 

1 

2 

3 

4 

5 
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4.  Do  you  feel  that  you  would  refer  to  us  in  the  future? 
□ Yes 

Why?  


□ No 
Why  not?_ 


5.  Comments: 


TTiank  you  for  your  iiii|>iitw 


Please  return  this  questionnaire  to: 
Lee  Jones 

Addiction  Assessment  Program 
99  Main  St. 

Anytown,  Ontario 
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L.  A Women’s  Day  Program 


Program:  The  program  is  a 28-day  intensive  treatment  program,  with  a 12-month  weekly 
aftercare  group  for  women  with  alcohol/drug  problems.  The  program  provides  education  and 
counselling  around  substance  abuse  issues  with  particular  focus  on  the  special  needs  of  women. 
The  28-day  program  operates  on  a closed  group  model  with  a new  group  of  10  women  starting 
every  six  weeks.  After  completing  the  day  program,  clients  join  one  of  the  open  ongoing 
aftercare  groups. 

The  agency  had  a core  of  volunteers  (many  of  them  former  clients),  and  they  were  able  to 
recruit  several  volunteers  to  help  with  many  aspects  of  the  evaluation. 

This  example  includes  quality  enhancement  questions  for  four  levels  of  evaluation: 


Part  1.  Client  Characteristics 

Part  2.  Outcome  Evaluation  - Program  Attendance  and  Completion 
Part  3.  Client  Characteristics  by  Program  Attendance  and  Completion 
Part  4.  Outcome  Evaluation  - Follow-Up. 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  moderate 


Part  1.  Client  Characteristics 


What  were  the  characteristics  of  clients  who  complete  xm  intake 
assessment  for  the  program  in  terms  of:  level  of  conmmption  cf  alcohol 
or  drugs;  whether  problem  substance  was  alcohol,  other  drugs,  or  both 
alcohol  and  other  drugs;  level  of  self-e^^  age;  and  child  care  needs? 


a.  Who  was  asking  the  question  and  why  did  they  want  this  information? 

The  program  had  been  operating  for  a number  of  years  and  the  staff  and  manager  had  the 
perception  that  some  of  the  characteristics  of  current  clients  differed  from  those  whom  the 
program  initially  targeted.  First,  although  the  program  was  initially  designed  for  women  who 
abused  alcohol,  staff  perceptions  indicated  that  drug  use  was  high  as  well.  There  was  also  an 
impression  that  the  average  age  of  clients  was  decreasing  and  that  child  care  needs  were 
increasing.  Self-esteem  had  always  been  a concern  of  the  program,  and,  while  initial  self-esteem 
levels  of  clients  did  not  appear  to  have  changed  from  previous  years,  self-esteem  level  was  a 
characteristic  that  the  program  staff  wanted  to  continue  to  monitor. 

The  first  stage  of  the  quality  enhancement  process  was  to  obtain  an  accurate  and  up-to-date 
description  of  the  client  population  to  ensure  that  services  were  in  place  to  address  current 
problems.  In  addition,  this  information  would  be  used  as  baseline  information  for  conducting 
outcome  evaluation  for  the  same  clients. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

The  assessment  procedures  already  included  most  of  the  information  (including  a self-esteem 
measure,  age,  and  information  on  children  and  child  care  needs).  As  part  of  the  baseline  data 
collection,  the  program  began  using  the  Timeline  Follow-Back  (TLFB)  procedure  to  obtain 
accurate  information  on  alcohol  use,  as  well  as  a drug  assessment  interview  for  drug  use.  These 
measures  are  described  in  greater  detail  at  the  end  of  this  part  of  the  example.  A volunteer  who 
was  experienced  in  data  analysis  spent  50  hours  reviewing  case  files  and  extracting,  entering, 
and  analyzing  the  information. 
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In  order  to  analyze  the  data,  a computer  and  statistical  software  were  required. 


c.  How  were  the  data  collected? 

The  data  were  collected  as  part  of  the  clinical  assessment.  To  address  the  quality  enhancement 
question,  the  following  measures  were  extracted  by  the  volunteer  from  the  assessment  data: 

(a)  Level  of  consumption  of  alcohol  or  drugs.  Variables  used  to  measure  this  included: 
maximum  number  of  drinks  per  day,  average  number  of  standard  drinks  per  day,  and 
number  of  days  using  drugs  other  than  alcohol  (variables  were  calculated  from  information 
obtained  using  the  Timeline  Follow-Back  procedure  and  the  drug  assessment  interview 
covering  the  six  months  prior  to  entering  treatment). 

(b)  Problem  substance  alcohol,  drugs,  or  both.  All  alcohol  and  drug  use  was  recorded  for  the 
six  months  prior  to  entering  the  program.  Problem  alcohol  use  was  defined  as  drinking 
five  or  more  drinks  on  one  occasion  at  least  once  a week  or  drinking  on  average  more 
than  14  drinks  per  week,  or  being  reported  by  the  client  as  a problem.  Problem  drug  use 
was  defined  as  using  the  drug  once  a week  or  more  frequently  during  the  six  month 
period,  or  any  use  defined  as  a problem  by  the  client. 

(c)  Self-esteem  was  measured  using  the  Rosenberg  Self-Esteem  Scale  (RSES)^ 

(d)  Age  was  measured  as  age  in  years  at  time  of  entry  into  the  program. 

(e)  Child  care  needs  were  assessed  using  three  variables:  whether  the  client  had  any 
dependent  children  (16  years  of  age  and  under)  living  with  her,  whether  any  of  these 
children  were  five  years  of  age  or  younger,  and  whether  the  client  reported  that  she  would 
have  difficulty  arranging  child  care  in  order  to  attend  the  program. 

The  first  step  was  to  prepare  the  data  for  computer  entry.  A summary  sheet  was  set  up  for 
extracting  information  from  the  assessment  forms  in  each  case  file.  Information  was  coded  for 
each  variable  as  shown  in  Figure  1. 


'The  Rosenberg  Self-Esteem  Scale  (RSES)  consists  of  10  statements  that  describe  overall  feelings  of  self-worth 
and  self-acceptance.  Respondents  indicate  their  agreement  with  each  statement  using  a 4-point  scale  ranging  from 
"Strongly  agree"  to  "Strongly  disagree"  (see  p.217).  The  RSES  data  that  was  produced  for  this  example  was 
inadvertently  based  on  a 5-point  rating  scale.  The  4-point  rating  scale  is  recommended  for  use  by  programs  in  their 
quality  enhancement  endeavours. 
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d.  How  were  the  data  analyzed? 

The  information  was  entered  into  the  computer  using  a simple  statistical  software  package:  EPI 
INFO.  Descriptive  statistics  (i.e.,  counts  and  percentages)  were  calculated  on  categorical 
variables,  such  as  whether  problem  substance  was  alcohol  and/or  drugs.  Average  scores  and 
ranges  of  scores  were  calculated  for  quantifiable  variables  (e.g.,  average  age  of  clients). 

Pie  graphs  illustrating  some  of  the  results  were  generated  using  Lotus  1-2-3  (see  Appendix  B). 


e.  What  did  they  find  out? 

During  a 24  month  period,  290  women  were  assessed.  As  shown  in  Table  1,  the  majority  of 
clients  (70%)  had  problems  with  both  alcohol  and  other  drugs  (see  also  pie  graph  shown  in 
Figure  2).  Clients  also  tended  to  have  low  self-esteem.  More  than  a third  of  clients  had  at  least 
one  child  aged  five  or  younger  and  about  a third  of  clients  reported  that  arranging  child  care 
would  be  a problem  (see  Figure  3). 
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Table  1.  Characteristics  of  the  290  Clients  Assessed 


Measure 

Results 

Substance  use: 

average  maximum  drinks  per  day 

11.0  (range  0 to  24) 

average  number  of  drinks  per  day 

5.3  (range  0 to  12) 

average  number  of  drug-using  days 

82.7  (range  0 to  182) 

Problem  substance: 

alcohol  only 

50  (17%) 

drugs  only 

39  (13%) 

alcohol  and  drugs 

201  (69%) 

Self-esteem: 

average  rating  (where  1 is  lowest  self-esteem  and  5 is 
highest  self-esteem) 

2.4  (range  1.2  to  4.5) 

Average  age: 

32.7  (range  18  to  65) 

Child  care  needs: 

number  of  clients  with  no  dependent  children 

93  (32%) 

number  of  clients  with  children  aged  6-16,  but  no 
children  under  6 

84  (29%) 

number  of  clients  who  had  at  least  one  child  aged  5 or 
younger 

113  (39%) 

number  of  clients  reporting  that  arranging  child 
care  would  be  a problem 

92  (32%) 

NOTE:  Average  number  of  drinks  and  number  of  drug-using  days  are  based  on  analyses  of  all  clients 
assessed  (including  those  who  did  not  report  problems  with  alcohol  or  did  not  have  problems  with  drugs). 

Among  those  who  reported  problem  use  of  alcohol,  the  average  number  of  drinks  per  day  was 
5.8  (range  1 to  12)  and  the  average  maximum  number  of  drinks  per  day  was  12.0  (range  3 to 
24).  Among  those  who  were  identified  as  having  problems  related  to  drug  use,  the  average 
number  of  days  when  drugs  were  taken  over  the  past  six  months  (182  days)  was  96.8  (range  12 
to  182).  The  most  frequently  used  drugs  were  marijuana/hash,  cocaine,  and  tranquilizers 
(benzodiazepines) . 
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Figure  2.  Proportion  of  Clients  Abusing  Alcohol  Only,  Drugs  Only,  or  Both 


Alcohol  and  Drug  (69.3%) 


Alcohol  only  (17.2%) 


Drugs  only  (13.4%) 


Figure  3.  Proportion  of  Clients  With  and  Without  Dependent  Children 


No  depend,  child.  (32.1%) 


At  least  one  child  <6  years  (39.0%) 


Women’s  Day  Program 


/.  How  were  the  results  used? 


214 


The  most  important  finding  was  that  most  clients  assessed  for  the  program  reported  problem  use 
of  both  alcohol  and  at  least  one  other  psychoactive  drug.  This  result  suggested  that,  although 
the  program  was  initially  seeing  mostly  alcohol  abusers,  current  clients  were  more  likely  to  be 
polydrug  users.  The  use  of  more  than  one  substance  raised  issues  of  cross-addiction  and  other 
complications  relevant  to  treatment.  Also,  the  social  context  of  alcohol  use  is  likely  to  be 
different  than  the  social  context  of  use  of  drugs,  such  as  prescribed  benzodiazepines  or  street 
drugs  such  as  cocaine.  Thus  the  program  needed  to  address  both  alcohol  and  drug  abuse  issues, 
as  well  as  ongoing  abuse  of  several  substances.  A decision  was  made  by  the  manager  and  staff 
that  the  priority  for  staff  training  would  be  education  regarding  use  and  abuse  of  both  illicit  and 
prescription  drugs  as  well  as  issues  related  to  polydrug  use. 

Since  the  results  indicated  that  average  alcohol  consumption  levels  varied  considerably  among 
the  clients,  the  program  manager  and  one  staff  member  decided  to  set  aside  time  to  review  the 
content  of  the  program  to  ensure  that  the  needs  of  both  heavier  and  lighter  alcohol  abusers  could 
be  addressed. 

As  is  common  among  women  who  present  for  substance  abuse  problems,  self-esteem  was 
generally  low.  The  program  already  focused  on  improving  self-esteem  in  order  to  help  clients 
in  their  recovery.  These  results  confirmed,  for  the  program,  this  focus  on  self-esteem. 

The  average  age  of  clients  attending  the  program  had  decreased  over  the  years  of  operation. 
Since  younger  women  are  more  likely  to  have  young  children  and  more  likely  to  use  illicit 
drugs,  this  decrease  in  age  of  the  clientele  was  consistent  with  the  increase  in  drug  abuse  and 
the  greater  amount  of  time  spent  by  the  program  on  helping  clients  to  resolve  child  care 
problems. 

While  child  care  was  not  an  issue  with  most  clients,  it  did  present  problems  for  almost  a third 
of  those  assessed.  Therefore,  the  program  staff  felt  that  they  needed  to  continue  to  monitor  child 
care  needs  in  order  to  ensure  that  child  care  was  not  a barrier  to  treatment.  One  staff  member 
was  assigned  the  task  of  investigating  day-care  options  in  the  community  for  program 
participants.  In  addition,  the  possibility  of  in-house  day-care  would  be  discussed  by  the  board 
and  the  alumnae  group. 
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Methodological y measurement,  and  ethical  issues: 

Methodological.  The  present  approach  to  monitoring  child  care  needs  may  actually 
underestimate  this  need  in  the  potential  client  population.  It  may  be  that  many  women  who 
would  be  suitable  for  the  program  are  not  referred  in  the  first  place  because  problems 
arranging  child  care  would  preclude  entering  the  program.  Thus,  it  may  be  that  most 
potential  clients  who  have  children  five  or  younger  are  already  being  screened  from  the 
program. 

Measurement.  Some  of  the  results  were  based  on  using  standardized  measures,  but  the 
categorization  of  problem  substance  (alcohol,  drugs,  or  both)  was  based  on  the  judgement 
of  the  assessment  worker  (using  the  designated  criteria).  These  criteria  provide  a handy  but 
rough  description  of  the  population.  For  example,  criteria  for  rating  whether  a client 
abused  drugs  did  not  distinguish  between  different  types  of  drugs  - thus,  weekly  use  of 
marijuana  was  classified  as  problem  use,  as  was  weekly  use  of  heroin  or  cocaine.  Since 
fairly  high  drug  use  was  identified  among  the  client  population,  the  program  might  adopt  a 
more  fine-grained  method  of  categorizing  drug  abuse  in  future  evaluations. 
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Measures  Used: 

Rosenberg  Self-Esteem  Scale 

This  scale  is  available  in  the  public  domain  and  is  reproduced  on  the  following 

page. 

Timeline  Follow-Back  Method 

The  Timeline  Follow-Back  method  is  a procedure  for  obtaining  information 
regarding  daily  alcohol  consumption,  using  a calendar  and  other  memory  aids.  Although 
the  method  takes  more  time  (10-30  minutes)  to  administer  than  some  other  assessments 
of  consumption,  it  is  considered  the  most  valid  measure  of  alcohol  consumption  currently 
available.  A description  of  how  to  use  this  method  is  available  in: 

A.  "A  Directory  of  Client  Outcome  Measures  for  Addictions  Treatment  Programs" 
(a  resource  book  available  from  the  Addiction  Research  Foundation,  1993) 

B.  "Timeline  Follow-Back.  A Technique  for  Assessing  Self-Reported  Alcohol 
Consumption"  by  L.C.  Sobell  and  M.B.  Sobell.  In  R.  Z.  Litten  & J.  Allen 
(Eds.),  Measuring  Alcohol  Consumption:  Psychological  and  biological  methods. 
New  Jersey:  Humana  Press,  1992. 


Drug  Use  Interview 

For  a description  of  approaches  to  conducting  a drug  use  interview  see: 

A.  A Directory  of  Client  Outcome  Measures  for  Addictions  Treatment  Programs" 
(a  resource  book  available  from  the  Addiction  Research  Foundation,  1993) 

B.  "Dimensions  of  Multiple  Drug  Use  and  Typology  of  Drug  Users"  by  D.A. 
Wilkinson,  G.M.  Leigh,  J.  Cordingley,  G.W.  Martin,  & H.  Lei,  British  Journal 
of  Addiction.  Vol.82.  pp. 259-273,  1987. 
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Rosenberg  Self-Esteem  Scale 


INSTRUCTIONS:  Please  circle  the  appropriate  number  for  each 

statement  depending  on  whether  you  strongly  agree,  agree,  disagree,  or 
strongly  disagree  with  it. 


Strongly 

agree  Agree  Disagree 

1 2 3 

1 . On  the  whole,  I am  satisfied  with  myself. 

2.  At  times  I think  I am  no  good  at  all. 

3.  I feel  that  I have  a number  of  good  qualities. 

4.  I am  able  to  do  things  as  well  as  most  other  people. 

5.  I feel  I do  not  have  much  to  be  proud  of. 

6.  I certainly  feel  useless  at  times. 

7.  I feel  that  I’m  a person  of  worth,  at  least  on  an 
equal  plane  with  others. 

8.  I wish  I could  have  more  respect  for  myself. 

9.  All  in  all,  I am  inclined  to  feel  that  I am  a failure. 

10.  I take  a positive  attitude  toward  myself. 


Strongly 

disagree 

4 

12  3 4 
12  3 4 
12  3 4 
12  3 4 
12  3 4 
12  3 4 

12  3 4 
12  3 4 
12  3 4 
12  3 4 


Scoring:  Sum  or  average  the  ratings  assigned  to  all  the  items  after  reverse  scoring  the  positively  worded  items. 
Total  scores  range  from  10  to  40.  Average  ratings  range  from  1 to  4.  Higher  scores  indicate  higher  self-esteem. 
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Level  of  difficulty:  easy 

Level  of  resources  needed:  moderate 


Part  2.  Outcome  Evaluation  - Program  Attendance  and  Completion 


What  proportion  cf  cHents  were  to  but  did  not  attend  the 

program? 

What  proportion  of  clients  whd  entered  the  program  atte 
What  proportion  of  clients  completed  a discharge  interview? 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 

The  program  manager  and  staff  wanted  to  know  more  about  attendance  at  and  dropout  from 
the  program.  While  some  dropout  is  inevitable,  the  program  manager  and  staff  were  concerned 
about  clients  who  were  assessed  but  did  not  show  for  treatment.  For  those  who  came  to 
treatment,  the  staff  felt  that  the  effectiveness  of  the  program  was  greatly  reduced  if  clients  did 
not  attend  regularly  or  dropped  out  without  completing  the  program.  The  first  step  in  addressing 
this  issue  was  to  assess  the  rate  of  attendance  and  program  completion  of  the  clients  who  had 
attended  the  program.  Attendance  and  program  completion  were  calculated  for  the  clients  who 
were  assessed  during  Part  1 of  the  evaluation. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

All  information  was  recorded  in  the  client’s  files.  A volunteer  spent  40  hours  extracting  and 
analyzing  the  information. 


c.  How  were  the  data  collected? 

A file  was  opened  for  each  client  assessed.  Whether  the  person  entered  the  program  was 
indicated  in  the  file,  as  well  as  daily  attendance  at  the  day  treatment  program,  and  whether  the 
client  completed  a discharge  interview.  Clients  who  attended  at  least  20  days  of  the  28-day 
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program  and  who  participated  in  the  discharge  interview  were  classified  as  "successful  program 
completers" . 


d.  How  were  the  data  analyzed? 

The  volunteer  added  the  data  to  the  existing  database  in  EPI  INFO  and  calculated  the  following: 

• proportion  of  clients  who  entered  the  program 

• proportion  of  clients  entering  the  program  who  attended  19  days  or  less 

• proportion  of  clients  entering  the  program  who  attended  20  days  or  more 

• proportion  of  clients  entering  the  program  who  completed  a discharge  interview 

• proportion  of  "successful  completers"  (i.e.,  attended  20  days  or  more  and 
attended  discharge  interview). 

Lotus  1-2-3  (see  Appendix  B)  was  used  to  prepare  a graphical  presentation  of  the  results. 

e.  What  did  they  find  out? 

Of  the  290  people  who  were  assessed  for  the  program,  70  (24%)  did  not  begin  the  program;  that 
is,  they  dropped  out  between  the  assessment  and  program  entry.  Of  the  remaining  220  who  did 
enter  the  program,  169  (77%  of  those  who  entered  treatment)  were  classified  as  "successful 
completers".  Figure  1 provides  a pie  chart  illustrating  the  distribution  of  persons  assessed  into 
three  groups:  did  not  attend  at  all  (non-attenders),  attended  but  did  not  successfully  complete 
the  program  (non-completers),  and  those  who  successfully  completed  the  program.  Average 
attendance  rates  of  all  who  entered  the  program  was  22  days.  Among  program  completers,  the 
average  attendance  rate  was  26  days. 
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Figure  1.  Proportion  of  Clients  Assessed  Who  Entered  and  Completed  Treatment 


Successful  completers  (58u3%) 


/.  How  were  the  results  used? 

The  results  indicated  that  the  largest  proportion  of  clients  who  dropped  out  did  so  before 
entering  the  program.  This  finding  suggested  to  the  program  manager  that  one  area  that  needed 
to  be  examined  was  why  people  who  were  assessed  for  the  program  chose  not  to  come.  This 
dropout  rate  reduced  the  program’s  efficiency  and  hence  failed  to  serve  clients  who  potentially 
could  benefit  from  attending  it.  A closer  examination  of  the  characteristics  of  these  clients  was 
planned  (and  is  reported  in  Part  3 of  this  example). 

Once  clients  entered  the  program,  retention  rates  and  attendance  were  fairly  high.  However, 
the  program  manager  also  felt  that  more  needed  to  be  known  about  non-completers,  particularly 
those  who  dropped  out  of  the  program  during  the  first  few  weeks.  In  addition,  the  program  is 
considering  whether  more  active  outreach  and  more  frequent  referrals  are  needed  for  those  who 
do  not  respond  well  to  the  program. 
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Level  of  difficulty:  moderate 
Level  of  resources  needed:  moderate 


Part  3.  Client  Characteristics  by  Program  Attendance  and  Completion 


Did  clients  who  dropped  out  b^re  or  dunng  treatment  differ  from  clients 
who  completed  the  prognm  on  iMtial  a^^  and  drug  usey  substances 
abused j initial  level  of  self-esteem,  arid  child  care  responsibilities  ? 


a.  Who  was  asking  the  question  and  why  did  they  want  this  information? 

The  program  manager  and  staff  wanted  to  better  understand  the  needs  of  clients  who  dropped 
out  before  or  during  treatment.  Using  the  data  that  had  already  been  collected  on  a number  of 
variables  addressed  at  the  initial  assessment,  it  was  possible  to  examine  the  differences  between 
those  who  completed  the  program  and  those  who  did  not.  This  information  might  be  used  to 
help  reduce  dropout  rates. 


b.  What  resources  were  needed  to  collect  and  interpret  the  information  ? 

An  experienced  volunteer  spent  three  hours  analyzing  data  that  had  already  been  entered  into  the 
computer  database. 


c.  How  were  the  data  collected? 

All  information  had  been  entered  into  the  computer  during  Parts  1 and  2 of  the  evaluation. 


d.  How  were  the  data  analyzed? 

The  volunteer  spent  three  hours  generating  comparison  scores  on  the  assessment  variables 
computed  in  Part  1 of  the  example  for  those  who  dropped  out  between  assessment  and  the 
beginning  of  the  day  treatment  program  ("non-attenders"),  those  who  attended  the  program  but 
were  not  classified  as  successful  completers  ("non-completers"),  and  those  who  attended  the 
program  20  or  more  days  and  attended  the  discharge  interview  ("successful  completers").  The 
volunteer  also  calculated  average  attendance  scores  for  all  clients  who  entered  treatment. 
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comparing  those  who  had  children  to  those  who  did  not,  and  comparing  those  who  had  at  least 
one  child  five  years  of  age  or  younger  to  those  who  had  no  children  or  whose  children  were  six 
or  older.  Average  scores  among  the  three  groups  were  compared  for  statistical  significance 
using  Analysis  of  Variance.  Statistical  significance  of  relationships  among  categorical  variables 
was  assessed  using  Chi-Squared  statistic. 


e.  What  did  they  find  out? 

As  identified  in  Part  2 of  the  evaluation,  of  the  290  people  who  were  assessed  and  referred  to 
the  program,  70  dropped  out  between  assessment  and  the  beginning  of  treatment,  169  were 
classified  as  successful  completers  and  51  attended  the  program  but  were  not  classified  as 
successful  completers. 

As  shown  in  Table  1,  the  results  indicated  that  successful  completers,  non-completers,  and  non- 
attenders  did  not  differ  significantly  in  terms  of  initial  levels  of  alcohol  use.  Successful 
completers,  however,  tended  to  use  drugs  less  often  than  non-attenders  and  non-completers. 
Successful  completers  also  tended  to  be  more  likely  than  non-attenders  and  non-completers  to 
have  problems  with  alcohol  only,  while  non-attenders  and  non-completers  were  more  likely  to 
abuse  both  alcohol  and  drugs. 

In  terms  of  self-esteem,  successful  completers,  non-completers  and  non-attenders  were 
significantly  different  on  initial  self-esteem,  with  successful  completers  highest  in  self-esteem. 

The  three  groups  did  not  differ  significantly  in  age,  but  there  was  a significant  difference  in  the 
percent  of  clients  in  each  group  who  had  dependent  children.  Clients  who  had  no  dependent 
children  were  more  likely  to  complete  the  program  while  clients  who  had  at  least  one  child  aged 
five  or  younger  were  more  likely  to  drop  out  before  treatment  or  to  enter  the  program  but  not 
complete  it.  Similarly,  those  who  had  children  attended  the  program  less  often  than  those  with 
no  children,  and  clients  who  had  at  least  one  child  aged  five  or  younger  showed  the  poorest 
attendance. 
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Table  1.  Initial  Assessment  Results  for  Successful  Completers,  Those  Who  Attended 
the  Program  But  Were  Not  Classified  as  Successful  Completers  (Non- 
completers) and  Those  Who  Dropped  Out  Between  Assessment  and  the 
Beginning  of  the  Program  (Non-attenders). 


All  assessed 
clients 
n=290 

Successful 

completers 

n=169 

Non- 

completers 

n=51 

Non- 

attenders 

n=70 

Substance  use: 

average  maximum  of  drinks  per  day 
(F  = 2.7,  df=289,  n.s.) 

11.0 

10.7 

10.0 

12.2 

average  number  of  drinks  per  day 
(F=2.5,  df=289,  n.s.) 

5.3 

5.0 

5.2 

5.9 

average  number  of  drug-using  days 
(out  of  182) 

(f=3.4,  df=289,  p<.05) 

82.7 

76.4 

90.0 

92.7 

Problem  substance: 

(X^=18.2,  df=4,  p<.01) 
alcohol  only 

50 

(17%) 

40 

(24%) 

2 

(4%) 

8 

(11%) 

drugs  only 

39 

(13%) 

27 

(16%) 

7 

(14%) 

5 

(7%) 

alcohol  and  other  drugs 

201 

(69%) 

102 

(60%) 

42 

(82%) 

57 

(81%) 

Self-esteem: 

average  self-esteem  score 
(where  1 is  lowest  self-esteem 
and  5 is  highest) 

(F=13.1,  df=289,  p<.0001) 

2.4 

2.6 

2.1 

2.2 

Average  age 
(F=1.6,  df=289,  n.s.) 

32.7 

33.4 

30.8 

32.3 

Child  care: 

(X^=75.8,  df=4,  p<.001) 

number  of  clients  with  no  dependent 
children 

93 

(32%) 

80 

(47%) 

5 

(10%) 

8 

(11%) 

number  of  clients  with  children  aged 
6-16,  but  none  under  6 

84 

(29%) 

53 

(31%) 

16 

(31%) 

15 

(21  %) 

number  of  clients  with  at  least  one 
child  aged  5 or  younger 

113 

(39%) 

36 

(21%) 

30 

(59%) 

47 

(67%) 
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The  results  indicated  that  having  problems  with  more  than  one  substance,  having  low  self- 
esteem, and  having  dependent  children  (especially  children  aged  five  and  under)  were  related 
to  dropping  out  and  poor  attendance.  These  results  suggested  that  more  needed  to  be  done  at 
the  assessment  stage  and  during  the  program  to  facilitate  treatment  for  those  with  low  self-esteem 
and  polydrug  problems.  The  program  manager  and  staff  are  experimenting  with  various 
methods  of  support  and  outreach  to  increase  the  retention  rates  of  these  clients.  In  addition,  the 
evident  impact  of  dependent  children  on  treatment  entry  and  attendance  suggests  that  the 
program  needs  to  institute  a systematic  method  of  helping  clients  deal  with  child  care 
emergencies.  The  alumnae  group  is  setting  up  a core  of  volunteer  babysitters  who  will  be 
available  to  help  out  in  emergencies.  In  addition,  various  options  for  facilitating  day  care  are 
being  explored  by  a subcommittee  of  board  members  and  alumnae.  As  new  clients  are  assessed, 
the  availability  of  child  care  support  will  be  discussed,  including  the  availability  of  volunteers 
for  emergency  babysitting. 


Women’s  Day  Program 


225 


Level  of  difficulty:  difficult 
Level  of  resources  needed:  heavy 


Part  4.  Outcome  Evaluation  - Follow-Up  Status  at  12  Months 


Did  the  clients  who  completed  the  program  decrease  their  use  of  alcohol 
and  drugs?  : , . ■ , , . ■ ■ ■ ■ ■ ■ ■ ■ ■ . ■ ■ , , . , ■ ■ 

Did  the  clients  who  completed  the  program  improve  their  self-esteem? 

Did  successfid  program  completers^^^s^^  improvements  than  non- 

completers  and  non-attenders? 

How  many  clients  attended  aftercare? 

What  was  the  average  length  of  time  that  clients  attended  aftercare? 


a.  Who  was  asking  the  questions  and  why  did  they  want  this  information? 

As  part  of  the  process  of  quality  enhancement,  the  program  began  to  collect  systematic  follow-up 
data  at  12  months  following  treatment.  Clients  were  telephoned  by  volunteers  every  three 
months  for  the  first  12  months  following  their  discharge  from  treatment.  At  the  12-month 
follow-up,  clients  completed  a more  extensive  reassessment  including  use  of  the  Timeline 
Follow-Back  procedure,  a drug-use  interview  and  the  Rosenberg  Self-Esteem  Scale.  The 
program  manager  and  board  hoped  to  use  follow-up  information  as  well  as  rates  of  attendance 
at  aftercare  as  part  of  an  evaluation  to  assess  the  extent  to  which  clients  were  being  helped  by 
the  program.  This  information  might  also  be  useful  in  identifying  areas  where  the  program 
needed  to  be  strengthened. 
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b.  What  resources  were  needed  to  collect  and  interpret  the  information? 

Volunteers  spent  400  hours  over  two  years  contacting  clients  and  conducting  the  12-month 
follow-up  interviews.  As  well,  private  fundraising  allowed  the  program  to  hire  a half-time 
researcher  who  co-ordinated  the  follow-up  interviews  and  did  the  data  entry  and  analysis.  A 
seven-member  program  advisory  committee  oversaw  the  project. 


c.  How  were  the  data  collected? 

Volunteers  attempted  to  contact  all  persons  who  were  assessed  for  the  program  even  if  they  did 
not  enter  or  complete  treatment.  For  clients  who  did  not  attend  treatment  up  to  discharge, 
follow-up  contacts  occurred  at  three-month  intervals  following  the  date  they  would  have  been 
discharged  from  the  program  had  they  completed  treatment.  A modified  version  of  the  initial 
assessment  form,  the  Rosenberg  Self-Esteem  Scale,  a drug-use  interview,  and  the  Timeline 
Follow-Back  procedure  were  administered  by  the  volunteer  interviewer  when  the  12-month 
follow-up  contact  was  made. 

The  aftercare  group  kept  attendance  records  and  these  records  were  reviewed  for  the  290  clients 
involved  in  the  evaluation.  Two  measures  were  extracted:  the  number  of  aftercare  sessions 
attended  and  the  number  of  days  between  discharge  from  the  program  and  the  last  aftercare 
meeting  attended. 


d.  How  were  the  data  analyzed? 

The  data  were  entered  into  the  computer  using  EPI  INFO.  Assessment  information  and 
information  regarding  program  entry,  attendance  and  completion  already  existed  in  the  data  file. 
Follow-up  data  were  analyzed  comparing  results  of  those  who  completed  the  program  and  those 
who  did  not.  Changes  in  scores  between  initial  assessment  and  12-month  follow-up  were 
calculated.  Analysis  of  variance  comparisons^  were  used  to  assess  the  statistical  significance 
of  the  differences  on  these  scores  between  successful  completers  and  non-completers.  Proportion 
of  clients  attending  aftercare,  average  number  of  sessions  attended  and  average  length  of  time 
in  aftercare  were  also  calculated. 

Line  graphs  comparing  initial  measures  with  follow-up  measures  were  generated  using  Lotus 
1-2-3  (see  Appendix  B). 


‘Since  there  were  only  two  groups  being  compared,  using  analysis  of  variance  is  equivalent  to  using  t-test, 
except  that  the  F-statistic  is  reported  rather  than  the  t-statistic.  The  F-statistic  is  the  more  general  form  that  could 
be  used  with  three  or  more  groups,  if  desired. 
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Follow-up  was  attempted  for  everyone  who  had  completed  the  initial  assessment,  including  those 
who  dropped  out  before  and  during  treatment.  Twelve-month  follow-up  was  completed  for  204 
clients  (70%  of  all  who  were  assessed).  The  rate  of  follow-up  success  was  higher  for  the  220 
who  entered  the  program  (168  [76 %]  were  located  at  follow-up),  and  higher  still  for  the  175 
clients  who  completed  the  program  (139  [82 %]  located  at  follow-up). 

Table  1 shows  the  12-month  status  on  drinks  per  day,  number  of  drug-using  days,  and  self- 
esteem of  clients  who  did  not  complete  the  program  or  who  dropped  out  before  entering  the 
program  compared  to  successful  program  completers.  Table  2 shows  the  average  change  scores 
on  these  variables  for  those  who  did  not  complete  the  program  compared  to  those  who  were 
successful  completers.  The  changes  in  status  for  non-completers  versus  completers  are  shown 
graphically  in  Figures  la  to  Id. 

The  data  indicated  that  successful  completers  were  doing  significantly  better  at  12-month  follow- 
up than  those  who  dropped  out,  both  in  terms  of  absolute  outcome  (Table  1)  and  in  terms  of 
extent  of  change  made  (Table  2 and  Figures  la  to  Id). 


Table  1.  Substance  Use  and  Self-Esteem  Measures  at  Follow-Up  (Program  Completers 
Compared  to  Non-completers/Non-attenders) 


Non-completers 
and  non-attenders 
n = 65 

Successful 
completers 
n = 139 

average  maximum  number  of  drinks  in  one  day 
(F=49.0,  df=203,  p<.0001) 

9.6 

4.3 

average  number  of  drinks  per  day 
(F  = 69.2,  df=  203,  p<. 0001) 

4.3 

1.4 

average  number  of  drug-using  days  (out  of  182) 
(F=41.6,  df=  203,  p<. 0001) 

51.5 

18.2 

average  self-esteem  score  (where  1 is  lowest  self- 
esteem and  5 is  highest) 

(F=  156.7,  df=203,  p<.0001) 

2.5 

3.8 
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Table  2.  Changes  in  Average  Scores  Between  Assessment  and  Follow-Up  (Program 
Completers  Compared  to  Non-completers/Non-attenders) 


Non-completers 
and  non-attenders 
n=65 

Successful 

completers 

n=139 

change  in  average  maximum  number  of  drinks  in  one 
day 

(F  = 24.6,  df=203,  p = .0001) 

-1.9 

-6.6 

change  in  average  number  of  drinks  per  day 
(F=28.2,  df=203,  p<.0001) 

-1.3 

-3.6 

change  in  average  number  of  drug-using  days  (out  of 

182) 

(F=4.2,  df=203,  p<.05) 

-42.9 

-58.9 

change  in  average  self-esteem  score  (where  1 is  lowest 
self-esteem  and  5 is  highest) 

(F  = 51.3,  df=203,  p<.0001) 

-f-0.4 

+ 1.2 

With  regards  to  attendance  at  aftercare,  most  of  the  successful  completers  attended  at  least  one 
aftercare  group,  with  an  average  of  four  sessions  attended  (see  Table  3).  The  average  duration 
of  clients’  attendance  at  aftercare  was  slightly  longer  than  one  month. 


Table  3.  Aftercare  Attendance  of  Program  Completers  (n=175) 


Measure 

Results 

number  of  clients  who  attended  at  least  one  aftercare  group 

165  (94.3%) 

average  number  of  aftercare  sessions  attended  (range  0-40) 

4 

average  length  of  time  people  continued  to  attend  aftercare 

35  days 

following  discharge  (range  0-300) 
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The  results  indicated  that  those  who  successfully  completed  the  program  improved  substantially, 
and  that  they  improved  significantly  more  than  those  who  did  not  complete  the  program. 
Although  this  did  not  provide  absolute  proof  of  the  program’s  effectiveness  (since  no  control 
group  was  used  in  the  evaluation),  it  did  provide  evidence  that,  for  the  most  part,  the  program 
was  meeting  its  goals  in  terms  of  client  change.  Therefore,  the  program  manager  decided  that 
any  changes  in  programming  at  this  time  would  focus  on  keeping  people  in  treatment. 

Dropout  from  aftercare  was  higher  than  expected.  Therefore,  the  program  staff  are  now 
considering  holding  an  aftercare  "refresher"  day  once  every  four  to  six  months  as  a means  by 
which  they  might  be  able  to  provide  aftercare  services  to  those  who  do  not  attend  on  a regular 
or  long-term  basis.  Clients  could  be  invited  to  the  "refresher"  day  when  they  are  telephoned  by 
volunteers  for  follow-up. 


Methodological,  measurement,  and  ethical  issues. 

Methodological.  While  the  results  indicated  that  program  completers  did  better  than  non- 
completers and  non-attenders,  these  findings  support  the  interpretation  that  the  program  is 
effective  but  do  not  provide  conclusive  evidence.  Program  completers  may  simply  have 
been  more  motivated  to  address  their  problems  and  thus  showed  better  results  than  drop- 
outs. To  formally  assess  the  program’s  effectiveness  would  have  required  the  random 
assignment  of  clients  to  a control  or  comparison  group  at  assessment.  The  program  was 
not  able  to  undertake  such  an  evaluation.  These  results  did  support  the  assumption  that  the 
program  was  reasonably  successful  with  regards  to  its  objectives  for  clients. 

Measurement.  To  keep  the  example  as  simple  as  possible,  change  scores  were  analyzed 
(in  a simple  two-group  one-way  ANOVA).  A more  usual  approach  would  be  to  use  a two- 
group  repeated  measures  ANOVA,  thereby  allowing  for  analysis  of  interactions. 

Ethical.  Because  the  design  of  the  study  required  that  clients  be  contacted  12  months  after 
completing  the  program,  clients  were  asked  to  consent  to  the  follow-up  at  the  time  that  they 
were  initially  assessed.  Volunteers  were  trained  in  conducting  follow-up  and  were  made 
aware  that  the  names  of  clients  and  other  information  to  which  they  had  access  was  to  be 
kept  strictly  confidential. 
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Agnew,  N.  McK.  & Pyke,  S.W.  (1991).  The  Science  Game.  Englewood  Cliffs,  N.J.: 
Prentice-Hall  Inc.  (285  pages). 

This  book  is  a fairly  easy-to-read  introduction  to  research  in  the  behavioural  sciences  for  people 
with  some  social  science  background.  Its  17  chapters  include  information  about  research  design, 
field  study  research,  survey  research,  descriptive  statistics,  basic  inferential  statistics,  sampling, 
measurement,  report  writing  and  ethics. 


Austin,  M.J.,  Cox,  G.,  Gottieb,  N.,  Hawkins,  J.D.,  Kruzich,  J.M.  & Rauch,  R.  (1982). 
Evaluating  your  agency^ s programs.  London:  Sage  Publications.  (192  pages). 

This  easy-to-read  book  discusses  program  evaluation  principles  and  practices  in  seven  chapters: 
1)  Why  should  agencies  evaluate  their  own  programs?  2)  Getting  ready;  3)  Getting  started; 
4)  Measuring  program  effort;  5)  Evaluating  program  effectiveness;  6)  Assessing  program 
efficiency;  7)  Program  evaluation  and  program  planning.  Although  the  examples  are  not 
addictions  specific,  this  is  a very  useful  guide  to  self-evaluation. 


Berk,  R.A.  & Rossi,  P.H.  (1990).  Thinking  about  program  evaluation.  Newbury  Park, 
California:  Sage  Publications.  (127  pages). 

This  book  is  fairly  theoretical.  The  authors  define  a number  of  evaluation  concepts  and 
approaches  to  evaluation  using  the  following  headings:  1)  What  this  book  is  about;  2)  Key 
concepts  in  evaluation  research;  3)  Designing  new  programs:  A chronological  perspective;  4) 
Examining  ongoing  programs:  A chronological  perspective. 


Briar,  S.  & Blythe,  B.J.  (1985).  Agency  support  for  evaluating  the  outcomes  of  social  work 
services.  Administration  in  Social  Work,  9(2),  25-36. 

This  brief  article  discusses  the  role  of  program  evaluation  in  service  agencies.  It  addresses  the 
use  of  assessment  tools  to  gather  information  about  client  characteristics.  It  also  includes 
discussion  of  the  role  of  the  management  information  system  in  program  evaluation. 
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California  Society  of  Addiction  Medicine.  (1991).  Recommendations  for  design  of  treatment 
efficacy  research  with  emphasis  on  outcome  measures.  California  Society  of  Addiction 
Medicine.  (4  pages). 

This  brief  publication  identifies  some  of  the  issues  to  be  examined  in  outcome  evaluations  and 
outlines  some  of  the  specific  variables  to  be  measured.  It  provides  a good  checklist  of  the 
elements  of  an  evaluation  study  and  some  of  the  possible  variables  to  consider. 


Canadian  Evaluation  Society.  (1989).  The  value  of  evaluation:  A statement  for  managers. 
Ottawa:  Canadian  Evaluation  Society.  (8  pages). 

"This  brief  statement  was  prepared  by  the  Canadian  Evaluation  Society  to  tell  managers  and 
other  decision  makers:  1)  what  to  expect  from  evaluation;  2)  how  evaluation  provides  credible, 
objective  information  about  the  results  of  existing  activities  and  the  potential  effectiveness  of 
proposed  new  initiatives;  3)  how  evaluation  adds  value  to  activities  through  better  understanding 
of  results  and  the  reasons  for  them;  and  4)  how  evaluation  supports  and  challenges  management 
decision  making  by  providing  better  information  about  the  links  between  mandate,  objectives, 
activities,  and  results."  (p.l)  Three  case  examples  are  used  to  illustrate  elements  of  the 
statement.  The  statement  is  printed  in  English  and  French  and  is  available  from:  The  Canadian 
Evaluation  Society,  309  St.  James  Street,  Ottawa,  Ontario  KIR  5M8. 


Casswell,  S.  & Duignan,  P.  (1989).  Evaluating  health  promotion:  A guide  for  health 
promoters  and  health  managers.  Auckland:  Department  of  Community  Health,  School  of 
Medicine,  University  of  Auckland.  (28  pages). 

This  brief  and  easy-to-read  guide  discusses  the  rationale  of  evaluations.  An  actual  dialogue  on 
the  issue  of  health  promotion  evaluation  is  quoted.  A number  of  evaluation  examples  are 
described,  and  ways  in  which  the  results  can  be  used  are  outlined.  The  guide  provides  a good 
introduction  to  evaluation  of  health  promotion  programs. 


Ciarlo,  J.A.,  Brown,  T.R.,  Edwards,  D.W.,  Kiresuk,  T.J.  & Newman,  F.L.  (1986). 
Assessing  mental  health  treatment  outcome  measurement  techniques.  Washington:  U.S. 
Department  of  Health  and  Human  Services.  (90  pages). 

This  document  discusses  the  use  of  client  outcome  measures  in  outcome  evaluations.  It  is  fairly 
detailed  and  technical.  While  it  is  long,  it  is  not  difficult  to  read  and  it  is  informative  for  the 
individual  wishing  to  explore  different  kinds  of  client  outcome  measures. 
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Commission  on  Accreditation  of  Rehabilitation  Facilities.  (1988).  Program  evaluation  in 
alcoholism  and  drug  abuse  treatment  programs.  Tucson:  Commission  on  Accreditation  of 
Rehabilitation  Facilities.  (50  pages). 

This  document  is  organized  into  nine  sections:  1)  Acknowledgements;  2)  Introduction  to 
evaluation;  3)  Program  evaluation  components;  5)  Performance  standards  and  weighting;  6) 
Client  descriptors  and  supplemental  measures;  7)  Data  collection;  8)  Utilization;  and  9) 
Appendix.  It  guides  the  reader  logically  through  the  rationale  behind  evaluations  and  then  the 
planning,  implementation,  and  utilization  of  evaluations  for  alcohol  and  drug  abuse  treatment 
programs.  Through  the  use  of  step-by-step  examples,  it  addresses  many  of  the  common 
concerns  within  such  settings. 


Ellis,  D.,  Reid,  G.  & Barnsley,  J.  (1990).  Keeping  on  track:  An  evaluation  guide  for 
community  groups.  Vancouver:  Press  Gang.  (79  pages). 

This  guide  is  organized  into  six  chapters:  1)  Evaluation  can  be  useful;  2)  How  to  prepare  for 
an  evaluation;  3)  Negotiating  for  an  evaluation  method;  4)  How  to  develop  an  evaluation  design; 
5)  How  to  collect,  analyze,  and  use  the  evaluation  data;  and  6)  Guidelines  for  self-evaluation. 
The  guide  is  based  on  a method  of  evaluation  called  participant-focused  evaluation.  The  guide 
is  well  organized  and  easy  to  read.  It  is  very  useful  for  the  novice- to-moderate  evaluator  who 
wishes  to  plan  and  carry  out  their  own  evaluation  using  this  type  of  approach. 


Fromberg,  R.  (Ed.).  (1987).  Monitoring  and  evaluation:  Alcoholism  and  other  drug 

dependence  services.  Chicago:  Joint  Commission  on  Accreditation  of  Healthcare 
Organizations.  (58  pages). 

This  guide  to  monitoring  and  evaluating  alcohol  and  drug  dependence  programs  includes  four 
chapters:  1)  Interpretation  of  the  standards  for  alcoholism  and  other  drug  dependence  services; 
2)  Monitoring  and  evaluation  of  the  quality  and  appropriateness  of  care;  3)  Monitoring  and 
evaluation  in  alcoholism  and  other  drugs  dependence  services;  and  4)  Resources.  The  focus  is 
on  the  concepts  of  "quality  of  care"  and  "quality  assurance."  It  is  logically  presented  in  simple 
language.  It  is  available  from:  Joint  Commission  on  Accreditation  of  Healthcare  Organizations, 
875  North  Michigan  Avenue,  Chicago,  Illinois  60611. 
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Graham,  K.,  et  al,  (1993).  A directory  of  client  outcome  measures  for  addictions  treatment 
programs.  Toronto:  Addiction  Research  Foundation.  (304  pages). 

The  directory  reviews  standardized  measures  for  client  outcome  in  three  main  areas:  1) 
Substance  abuse  - including  alcohol  and  drug  consumptions,  consequences  of  use,  drinking  and 
drug-using  situations,  self-efficacy  and  coping;  2)  Other  life  areas  - including  psychological 
well-being,  physical  health  and  cognitive  functioning,  social  and  leisure  activities,  legal,  financial 
and  work  status;  and  3)  Treatment  response  - including  client  satisfaction  and  feedback. 


Hawkins,  J.D.  & Nederhood,  B.  (1987).  Handbook  for  evaluating  drug  and  alcohol 
prevention  programs;  Staff /team  evaluation  of  prevention  programs  (STEPP).  Washington: 
U.S.  Department  of  Health  and  Human  Services.  (113  pages). 

This  handbook  is  a step-by-step  guide  to  planning,  implementing,  and  using  the  results  of  an 
evaluation.  The  process  of  evaluation  is  defined  in  six  steps:  1)  Choosing  an  evaluation 
question;  2)  Designing  an  evaluation;  3)  Designing  measurement  instruments;  4)  Building  a data 
collection  plan;  5)  Analyzing  the  data;  and  6)  Reporting  the  findings.  A checklist  is  provided 
at  the  beginning  of  the  handbook  to  guide  the  reader  through  the  manual.  This  book  is  detailed, 
comprehensive  and  easy  to  read. 


Hersey,  J.C.  (1979).  "Dirty"  research  in  "real"  places:  A practitioner’s  guide  to  program 
evaluation  in  the  human  services.  Evaluation  and  Program  Planning,  21,  153-157. 

This  article  focuses  on  quality  improvement  of  programs  using  evaluation.  It  uses  simple 
language  and  presents  a logical  process  of  developing,  performing,  and  using  the  results  of  an 
evaluation. 


Hudson,  J.,  Mayne,  J.  & Thomlison,  R.  (Eds.).  (1992).  Action-oriented  evaluation  in 
organizations:  Canadian  practices.  Toronto,  Ontario:  Wall  & Emerson.  (340  pages). 

This  edited  book  provides  a useful  description  of  the  variety  of  approaches  that  can  be  used  in 
evaluation. 


Koosis,  D.J.  (1977).  Statistics  (2nd  Edition):  A self -teaching  guide.  NY:  John  Wiley  & 
Sons. 

This  is  a ‘how  to’  book  for  the  general  student  who  wants  to  learn  basic  statistical  concepts  and 
procedures.  It  assumes  only  a limited  knowledge  of  algebra  and  shows  how  analyses  can  be 
done  using  a hand  calculator. 
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Leukefeld,  C.G.  & Bukoski,  W.J.  (1991).  Drug  abuse  prevention  evaluation  methodology: 
A bright  future.  Journal  of  Drug  Education^  21(3),  191-201. 

This  article  provides  a brief  synopsis  of  drug  abuse  prevention  evaluation.  Emphasis  is  on 
methodological  issues  of  evaluation,  such  as  instrumentation,  control/comparison  groups, 
intervention  specificity,  clarifying  outcome  variables,  replication  issues,  and  measuring 
immediate  and  long-term  effects.  Some  specific  evaluation  questions  are  identified  as  well. 


Linney,  J.A.  & Wandersman,  A.  (1991).  Prevention  plus  III:  Assessing  alcohol  and  other 
drug  prevention  programs  at  the  school  and  community  level.  A four-step  guide  to  useful 
program  assessment.  Rockville,  MD:  Office  for  Substance  Abuse  Prevention.  (461  pages). 

This  manual  is  specifically  designed  to  be  used  by  school  and  community  personnel  who  are 
unskilled  in  program  evaluation.  It  provides  many  worksheets  and  examples  for  the  various 
components  of  evaluation:  identifying  goals  and  desired  outcomes;  process  assessment;  outcome 
assessment;  and  impact  assessment,  as  well  as  40  assessment  measures.  This  manual  can  be 
obtained  from:  The  National  Clearinghouse  for  Alcohol  and  Drug  Information,  P.O.  Box  2345, 
Rockville,  MD.,  20847. 


McDermott,  F.,  Pyett,  P.  & Hamilton,  M.  (1991).  Evaluate  yourself:  A handbook  for 
alcohol  and  other  drug  treatment  agencies.  Melbourne,  Australia:  National  Campaign 
Against  Drug  Abuse.  (52  pages). 

The  process  of  developing,  implementing  and  using  the  results  of  an  evaluation  are  outlined  in 
three  sections:  1)  Doing  the  evaluation;  2)  Decisions;  3)  Methods.  The  book  also  includes  a 
bibliography  and  glossary  of  terms.  The  focus  of  the  book  is  on  using  informal  evaluation  as 
part  of  ongoing  program  development. 


National  Institute  of  Mental  Health.  (1979).  Evaluation  in  practice:  A sourcebook  of 
program  evaluation  studies  from  mental  health  care  systems  in  the  United  States.  Rockville, 
MD:  U.S.  Department  of  Health  and  Human  Services.  (288  pages). 

This  is  a collection  of  real  evaluation  examples  from  various  mental  health  care  systems 
throughout  the  United  States.  The  examples  are  presented  using  the  following  categories:  1) 
Needs  assessment;  2)  Patterns  of  use;  3)  The  four  A’s  - acceptability,  availability,  accessibility, 
awareness;  4)  Client  outcome;  5)  Community  impact;  6)  Indirect  services:  consultation, 
education  and  inter-agency  relationships;  7)  Cost,  cost  outcome,  and  cost  effectiveness;  8) 
Quality  assurance;  and  9)  Mental  health  program  evaluation  in  the  year  2000. 
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Newman,  F.L.,  Hunter,  R.H.  & Irving,  D.  (1987).  Simple  measures  of  progress  and 
outcome  in  evaluation  of  mental  health  services.  Evaluation  and  Program  Planning,  10, 
209-218. 

This  article  focuses  on  the  role  of  outcome  measures  in  evaluation  and  on  the  importance  of 
selecting  appropriate  measures.  As  well,  the  article  identifies  a number  of  issues  that  need  to 
be  taken  into  consideration  when  using  outcome  measures. 


Nutter,  R.W.  & Weiden,  T.D.  (1988).  Program  evaluation  and  quality  assurance:  A 
reconciliation.  The  Social  Worker,  56(1),  18-22. 

This  article  describes  the  differences  between  program  evaluation  and  quality  assurance  and  the 
importance  of  using  both  to  improve  the  delivery  and  effectiveness  of  programs.  It  identifies 
the  specific  issues  that  can  be  addressed  by  quality  assurance  and  by  program  evaluation. 


Offield,  J.C.  (1989).  Quality  assurance  through  program  evaluation.  EAP  Digest,  Spring 
1989,  25-29. 

The  focus  of  this  article  is  on  the  evaluation  methods  used  to  ensure  that  Employee  Assistance 
Programs  are  doing  what  they  were  intended  to  do.  Specifically,  ways  of  evaluating  the 
following  issues  are  discussed:  utilization  of  services;  services  provided;  characteristics  of 
participants  vs.  non-participants;  grievance  mediation  services;  and  cost-benefit  analysis. 


Ogborne,  A.C.  & Gavin,  M.T.  (1990).  Quality  assurance  in  substance  abuse  treatment: 
Guidelines  for  practitioners.  Toronto:  Addiction  Research  Foundation.  (36  pages). 

This  document  focuses  on  the  quality  assurance  process  and  related  concerns.  The  first  section 
introduces  the  concept  of  quality  assurance.  Substance  abuse  treatment  programs  are  then 
discussed  in  terms  of:  1)  Management  structure;  2)  Program  and  services;  3)  Program 
evaluation;  4)  Personnel/staffing;  5)  The  quality  assurance  process;  and  6)  Further 
considerations  of  standard  setting. 


Posavac,  E.J.  & Carey,  R.G.  (1992).  Program  evaluation:  Methods  and  case  studies.  4th 
ed.  Toronto:  Prentice-Hall.  (332  pages). 

This  is  a comprehensive,  fairly  technical  guide  to  program  evaluation.  It  is  divided  into  six 
parts: 

Part  I.  Overview  of  program  evaluation  - 1)  Program  evaluation  and  organizations;  2) 
Planning  an  evaluation;  3)  Selecting  criteria  and  setting  standards;  4)  Measurement  principles 
and  tools;  5)  Ethical  standards  of  conducting  program  evaluations; 
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Part  IL  Evaluation  in  program  planning  and  monitoring  - 6)  Assessment  of  need;  7) 
Monitoring  the  operation  of  programs; 

Part  III.  Quantitative  evaluation  of  outcome  - 8)  Non-experimental  approaches  to 
outcome  evaluation;  9)  Quasi-experimental  approaches  to  outcome  evaluation;  10)  Analysis  of 
causes  of  change; 

Part  IV.  Additional  approaches  to  program  evaluation  - 11)  Analysis  of  costs  and 
outcomes;  12)  Qualitative  evaluation  of  need,  process,  and  outcome; 

Part  V.  Effective  application  of  findings  - 13)  Evaluation  reports:  interpreting  and 
communicating  findings;  14)  Favourable  evaluation  climate:  how  to  encourage  utilization; 

Part  VI.  Case  Studies. 


Rossi,  P.H.  & Freeman,  H.E.  (1989).  Evaluation:  A systematic  approach,  4th  ed.  Beverly 
Hills:  SAGE  Publications.  (496  pages). 

This  is  the  fourth  edition  of  a popular  text  on  classical  approaches  to  evaluation.  It  consists  of 
nine  chapters:  1)  Programs,  policies,  and  evaluations;  2)  Diagnostic  procedures;  3)  Tailoring 
evaluations;  4)  Program  monitoring  for  evaluation;  5)  Strategies  for  impact  assessment;  6) 
Randomized  designs  for  assessment;  7)  Non-randomized  designs  for  impact  assessment;  8) 
Measuring  efficiency;  and  9)  The  social  context  of  evaluation.  Each  chapter  begins  with  a 
summary  of  the  key  concepts  to  be  discussed. 


Rush,  B.  & Ogborne,  A.  (1991).  Program  logic  models:  Expanding  their  role  and 
structure  for  program  planning  and  evaluation.  The  Canadian  Journal  of  Program 
Evaluation,  6 (2),  93-105. 

This  article  introduces  the  concept  of  the  program  logic  model  and  discusses  the  use  of  such  a 
model  in  program  planning  and  evaluation. 

Rutman,  L.  (1980).  Planning  useful  evaluations:  Evaluahility  assessment.  Beverly  Hills: 
Sage  Publications.  (208  pages). 

This  book  consists  of  six  chapters:  1)  The  need  for  evaluability  assessment;  2)  Factors  affecting 
program  evaluability;  3)  Evaluability  assessment:  program  analysis;  4)  Evaluability 
assessment:  analyzing  the  feasibility  of  achieving  the  evaluation’s  purpose(s);  5)  Enhancing 
program  evaluability;  and  6)  Evaluability  assessment  and  evaluation  planning:  questions  and 
benefits.  This  book  provides  a useful  discussion  of  issues  that  should  be  taken  into  consideration 
before  mounting  a large  scale  evaluation. 
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Sommer,  R.  & Sommer,  B.B.  (1980).  A practical  guide  to  behavioral  research:  Tools  and 
techniques.  New  York:  Oxford  University  Press. 

This  book  describes  basic  research  techniques  in  an  easily  understood  format  with  lots  of 
examples. 


Spicer,  J.,  Owen,  P.  & Levine,  D.  (1983).  Evaluating  Employee  Assistance  Programs:  A 
sourcebook  for  the  administrator  and  counsellor.  Center  City,  MN:  Hazelden  Foundation. 
(89  pages). 

This  is  a comprehensive,  short,  and  easy-to-read  guide.  It  includes  two  sections:  1)  What  is 
evaluation?  and  2)  How  to  evaluate  EAPs.  It  guides  the  reader  step-by-step  through  an  actual 
EAP  evaluation.  The  second  section  includes  discussion  of  research  methods,  needs  assessment, 
process  evaluation,  and  impact  evaluation. 


U.S.  Department  of  Health  and  Human  Services,  Public  Health  Service.  (1989).  Making 
health  communication  programs  work:  A planner's  guide.  Washington:  National  Institutes 
of  Health,  (pp.  63-71.) 

Two  chapters  in  this  guide  discuss  the  basic  considerations  involved  in  evaluating  a program. 
As  well,  elements  of  evaluation  and  the  utilization  of  the  results  of  an  evaluation  are  examined. 
These  chapters  provide  a good  summary  of  the  types  of  issues  to  address  when  considering  an 
evaluation. 


Williamson,  J.W.  (1978).  Formulating  priorities  for  quality  assurance  activity: 
Description  of  a method  and  its  application.  JAMA^  238(7),  631-637. 

This  article  discusses  issues  that  need  to  be  taken  into  consideration  when  planning  an  evaluation 
study.  It  guides  the  reader  through  a process  of  priority  setting  to  clarify  evaluation  objectives 
in  order  to  provide  a clear  starting  point  for  evaluation  planning. 
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Appendix  A.  Using  EPI  INFO 


This  appendix  is  organized  into  three  sections:  1)  general  information  on  features  of  EPI  INFO 
5.0  as  described  in  a recent  release  of  EPI  INFO  Info  (which  also  includes  an  EPI  INFO  order 
form);  2)  instructions  for  installing  EPI  INFO;  and  3)  step-by  step  instructions  for  entering  and 
analyzing  data  using  EPI  INFO  (based  on  the  hypothetical  data  used  in  Example  H from  the 
casebook).  Additional  information  regarding  some  EPI  INFO  functions  and  some  helpful  hints 
for  using  EPI  INFO  are  included  in  this  last  section. 

This  appendix  provides  a basic  overview  of  EPI  INFO  to  help  you  try  out  the  software  package. 
If  EPI  INFO  appears  to  be  suitable  for  the  data  analysis  needs  of  the  your  program,  the  manual 
should  be  purchased. 


LSD,  Incorporated, 

Telephone  (404)  469-4098 


2075  A WesI  Park  Place,  Slone  Mountain,  GA.  087 
Fax  (404)  4^9-0681 
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Appendix  A.l.  - Recent  Release  of  EPI  INFO 
(Reproduced  with  permission  from  Trina  Peavy,  USD,  Incorporated) 


Epi  Info  Info 


WHAT  IS  EPI  INFO,  VERSION  5? 

Epi  Info  is  a series  of  microcomputer  programs  produced  at 
the  Centers  for  Disease  Control,  Atlanta,  and  the  World 
Health  Organization,  Geneva,  for  handling  epidemiologic 
rfaia  in  questionnaire  format  and  for  organizing  study  designs 
and  results  into  text  that  may  form  part  of  written  reports.  A 
questionnaire  can  be  set  up  and  processed  in  a few  minutes, 
but  Epi  Info  can  also  form  the  basis  for  a powerful  disease 
surveillance  system  database  with  many  files  and  record 
types.  It  includes  the  features  most  used  by  epidemiologists 
in  statistical  programs  such  as  SAS  or  SPSS  and  database 
programs  like  dBASE  combined  in  a single  system  that  may 
be  freely  copied  and  given  to  friends  and  colleagues. 

With  Epi  Info,  Version  5,  you  can: 

✓Do  word  processing. 

i/Typc  a questionnaire  of  up  to  500  lines  on  the  screen,  or  in 
your  favorite  word  processor. 

✓Without  programming,  begin  entering  data  in  seconds. 

✓ If  desired,  add  range  checking,  legal  values,  automatic  cod- 
ing, skip  patterns,  and  repeat  fields  to  the  entry  process. 

✓Enter,  find,  and  edit  data  in  files  containing  up  to  billions  of 
records,  if  enough  disk  space  exists. 

✓Produce  lists,  frequencies,  and  multidimensional  tables. 

✓Do  chi  square,  Mantel-Haenszel  and  Fisher  exact  tests,  rel* 
alive  risk,  stratified  relative  risk;  confiderK*  limits  by  the 
Cornfield,  Robins-Grcenland-Breslow,  Greenland-Robins, 
and  exact  methods;  means,  one-way  ANOVA  and  Kruskal- 
Wallis  tests,  multiple  linear  regression,  matched  case-coniol 
analysis,  sample  size,  and  chi  square  for  trend. 


✓Use  RECODE,  IF,  and  SELECT  statements  to  manipulate 
records  and  variables. 

✓Write  programs  to  allow  complex  operations  or  repeated 
use  by  clerical  personneL 

✓Make  histograms,  bar.  line  and  pie  charts,  and  scatter  plots 
from  within  the  ANALYSIS  program. 

✓Import  fixed-length,  comma-delimited,  dBASE  or  Lotus  1- 
2-3  files.  Use  dBASE  files  directly  in  ANALYSIS. 

✓Export  files  in  12  different  popular  formats  or  process  files 
and  create  new  files  of  results. 

✓Create  relational  file  systems,  linking  several  different 
questkxmaiie  files  during  data  entry  and  again  for  ANALY- 
SIS. 

✓Update  files  by  merging  in  revised  records. 

✓Create  index  files  for  rapid  searching  of  large  sets. 

✓Perform  anthropometric  calculations  using  international  ref- 
erence standards. 

✓Develop  customized  reports  in  your  own  formaL 

✓Program  the  word  processor  to  perform  complex  tasks  auto- 
matically. 

The  programs  included  in  Epi  Info  are: 

EPI  - The  main  menu  that  allows  access  to  other  programs. 

EPED  - A word  processor  for  creating  questionnaires  or  for 
general  use.  A special  facility  called  EPIAJD  in  EPED  runs 


programs  ihai  assist  m text  preparauon  and  m the  design  of 
epidemiologic  investigations. 

ENTER  - Produces  a data  file  auiomatically  from  a ques- 
uonnairc  created  in  EPED  or  another  word  processor.  Man- 
ages entry  of  data  from  a questionnaire  into  a disk  file.  Al- 
lows revision  of  file  format  even  after  records  have  been 
entered. 

ANALYSIS  " Produces  lists,  frequencies,  cross  tabulations, 
and  a variety  of  other  results  from  Epi  Info  or  dBASE  files. 
Appropriate  epidemiologic  statistics  accompany  the  tables. 
Records  may  be  selected  or  sorted  using  defined  variables, 
statements,  and  mathematical  and  logical  operations. 
Graphing,  complex  report  formatting,  generation  of  new  data 
sets  for  testing,  and  a programming  language  are  included  in 
ANALYSIS.  Data  items  spread  over  several  files  can  be 
linked  and  analyzed  as  a single  unit 

CHECK  — Sets  up  ranges,  legal  values,  automatic  coding, 
and  skip  patterns  for  the  ENTER  program,  if  desired.  Mathe- 
matical and  logical  operations  between  fields,  complex  jump 
patterns,  access  to  several  files  in  the  same  entry  process,  and 
use  of  custom  programs  provided  by  the  user  are  also  support- 
ed by  CHECK. 

STATCALC  - Calculates  statistics  from  table  values  entered 
from  the  keyboard.  STATCALC  performs  calculations  for 
single  and  stratified  2-by-2  tables,  sample  size,  and  single  and 
stratified,  chi  square  for  trend  using  the  Mantel  extension  of 
the  Mantel-Haenszel  procedure. 

CONVERT  - Converts  data  files  from  Epi  Info  into  12  other 
formats  for  a variety  of  database  and  statistics  programs. 

IMPORT  - Brings  in  files  from  other  systems  so  that  they 
can  be  used  in  Epi  Info. 

MERGE  ~ Merges  files  produced  from  questionnaires  with 
the  same  or  different  formats.  This  allows  combining  data 
files  entered  on  different  computers,  and  ufxladng  of  previ- 
ous rtctxds  using  data  frtxn  new  entries. 

VALIDATE  - Compares  two  Epi  Info  files  entered  by  dif- 
ferent operators  and  reports  any  differences. 

Help  files  — Much  of  the  material  in  the  reference  section  of 
the  manual  is  available  in  help  files  accessed  by  pressing 
<F1>  from  within  Epi  Info  programs. 


Sample  Programs  -Tao  complete  sample  surveillance  sys- 
tems. files  from  two  epidemic  investigations  and  a system  for 
anihropomeinc  calculations  in  nuiniional  surveys,  are  provid- 
ed as  examples  of  Ept  Info  use.  There  arc  numerous  other 
sample  data  files,  and  many  programming  examples  are  pro- 
vided in  the  manual. 

Tutorials--  Complete  interactive  tutorials  are  provided  to  in- 
troduce the  features  of  EPED  and  ANALYSIS. 

Epi  Info,  Version  5,  requires  an  IBM-compatible  microcom- 
puter running  the  PC-DOS  or  MS-DOS  operating  system 
(Version  2.0  or  higher),  512,000  bytes  of  raiKlom-access 
memory  (RAM),  and  at  least  one  floppy  disk  drive.  A graph- 
ics adapter  board  is  necessary  to  prcxluce  graphs.  Adapters 
using  the  Hercules  or  IBM  monochrome,  EGA,  VGA. 
IBM3270.  IBM8514,  and  AT&T  standards  are  supported.  A 
printer  is  needed  to  produce  printed  copies  of  the  results; 
graphs  can  be  printed  with  IBM-  or  Epson-compatible  print- 
ers or  with  Hewlett-Packard -compatible  laser  printers  or  plot- 
ters. For  most  effective  use  of  Epi  Info  with  large  data  sets,  a 
hard-disk  drive,  640  K of  memory,  and  a color  graphics  board 
and  monitor  are  recommended. 

Data  files  can  consist  of  as  many  records  as  DOS  and  your 
disk  storage  can  handle  (billions).  A questionnaire  can  have 
up  to  500  lines  or  approximately  20  screens.  The  number  of 
variables  is  not  limited,  except  that  they  must  fit  in  the  500 
lines.  The  maximum  length  for  a text  variable  is  80  charac- 
ters. The  total  length  of  variables  in  one  file  must  not  exceed 
2048  characters.  The  programs  in  Epi  Info  require  less  than 
two  megabytes  of  disk  space. 

Versions  1-3  of  Epi  Info  were  produced  in  the  Epidemiology 
Program  Office  of  the  Centers  for  Disease  Control  (CDQ. 
Version  4 was  produced  and  distributed  on  an  experimental 
basis  by  the  Surveillance,  Forecasting,  and  Impact  Assess- 
ment Unit,  Global  Programme  on  AIDS,  World  Health  Or- 
ganization (WHO),  Geneva,  Switzerland,  in  collaboration 
with  CDC.  Version  5,  the  current  version,  is  a joint  CDC/ 
WHO  effort. 

The  programs  are  made  available  by  WHO  and  CDC  and  are 
not  copyrighted.  Making  copies  for  others  is  permitted  and 
encouraged.  Please  provide  access  to  copies  of  the  manual  if 
you  distribute  the  programs. 


WHAT’S  NEW  IN  VERSION  5? 

For  users  familiar  'Aiih  Version  3,  ihc  new  feaiures  in  Version 
5 are; 

✓True  missmg  value  indicators 
✓Pixed  decimal  fields 
i/Pufopean  date  fields  (dd/mm/yy) 

✓New  mathematical  and  siring  functions:  RND,  RAN,  LOG, 
EXP,  LN.TRIM 

✓BROWSE  and  UPDATE  commands  in  ANALYSIS  for 
viewing  and  modifying  data  sets. 

✓Linear  regression  with  up  to  15  variables 

✓Direct  analysis  of  dBase  11  and  dBase  III  files  without  us- 
ing IMPORT. 

✓Creadon  of  sample  data  sets,  using  the  GENERATE  com- 
mand 

✓A  revised  users'  manual  with  many  sample  programs 

✓Histograms,  bar  charts,  line  chans,  pie  chans,  and  scatter 
plots  with  regression  lines  within  ANALYSIS. 

✓Easy  revision  of  data  files 

✓Files  of  up  to  four  billion  records,  if  disk  space  allows 

✓LIST  • to  list  all  fields  automatically 

✓Defining  of  fields  in  ANALYSIS  as  in  quesuonnaires,  us- 
ing the  standard  Geld  indicators,  such  as  MM 

✓Sample  size  calcubdons 

✓Chi  square  for  trend  using  the  Mantel  extension  with  single 
and  stratified  tables 

✓Analysis  of  case-control  studies  with  matching 


✓Interacuve  programs  for  making  questionnaires.  EPI  I me- 
mos, and  study  design  desenpuons  m a new  facility  for  inter- 
acuve  text  generation  called  EPI.AID. 

✓Improved  editor  (EPED)  viih  pagination,  better  and  faster 
formatung,  user-defined  coiors,  and  additional  commands. 

✓Programming  features  in  EPED  allow  interactive  text  crea- 
tion (EPIAED) 

✓Complex  report  formatting  in  ANALYSIS  using  report  for- 
mat files 

✓Mathematical  operations  and  comparisons  among  multiple 
fields  during  data  entry 

✓Loops  and  subroutines  in  ANALYSIS  programs 

✓Searching  on  combinations  of  fields  in  ENTER 

✓Custom  routines  for  data  entry  written  by  the  user 

✓Automatic  numbering  of  questionnaire  records 

✓Pop-up  windows,  special  highlighting,  and  other  features 
for  entry  forms 

✓Relational  linking  of  several  files  in  ANALYSIS  using  the 
RELATE  command 

✓Relational  data  entry,  switching  automatically  between  files 
when  necessary.  A database  can  contain  files  for  demogr!q)h- 
ic  information,  hepatitis,  enteric  disease,  and  other  subjects, 
for  example,  each  with  a different  quesdonrtaire  . 

✓Indexed  file  searches  for  rapid  retrieval  of  records 

✓A  REPORT  command  that  allows  customization  of  tables 
and  other  reports 

✓Ability  to  save  prefererKx  settings  in  ANALYSIS 

✓Provision  for  non-English  character  sets. 

✓Compatibility  with  Version  3 data  files.  Programs  written 
for  Version  3 require  only  a few  changes  for  Version  5.  Ver- 
sion 5 data  files  should  not  be  used  in  Version  3 of  Epi  Info. 
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✓Tutorials  in  EPED  and  ANALYSIS 


i USD,  Incorporated 
I 2075-A  West  Park  Place 
I Stone  Mountain,  GA  30087  (U.S.A.) 
I (404M69-4098 

I FAX  (404)  469-0681 

1 

I 

I 

i 

i 

i 
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I 

i 

1 

* EPI-INFO  ORDER  FORM 

I VERSION  5.01  B COMES  WITH  FOUR  5 74'  DS/OD  AND  TWO  3 '/z  DSjDD  IBM  COMPATIBLE  DISKEHES  AND  A 384  PAGE  PRINTED  MANUAL 


LAST 

NAME  FIRST MIDDLE  ___ 

CO.  NAME 

ADDRESS 

ADDRESS 

CITY/PROVTNCE STATE  

COUNTRY POSTAL  CODE 

) - FAX#( ) 

IS  THIS  A RESIDENTIAL  ADDRESS?  (Y)  (N) 

CHECK  ENCLOSED  [ ] OR  PLEASE  BILL  MY:  MASTERCARD  [ ] VISA  [ ] AMERICAN  EXPRESS  [ 

NAME  ON  CARD 

CREDIT  CARD  NUMBER 

DESCRIPTION  QTY  COST  TOTAL 

COlyPLETE  SET  (DJSKS  & MANUAL)  XS38.M-  

DISKETTES  ONLY  X $15.00-  

MANUAL  ON  DISK  X $15.00-  

VIDEO  TAPE  NTSC  X $22.00  - 

PAL  X $2ZOO  - 

SECAM X $22.00  - 

VIDEO  ♦ MANUAL  ♦ DISKETTES  ^ X $55.00  - 


EXP.  DATE 


Credit  Card,  Check,  Money 
Order,  or  U.S.  Federal  Govern- 
ment purchase  order  required 
unless  approved  in  advance. 

[ ] Check  or  money  order 
enclosed,  payable  to  U.S.D.,  Inc. 
(Must  be  drawn  on  U.S.  Bank) 


SHIP  TO: 

sinEET 

ADDRESS 

ONLY! 

NO  P.O, 
BOXES!!!! 

TELEPHONE  ( 


FT^EIGHT  PER  MANUAL  (OUTSIDE  CONTINENTAL  U.S.A.) 

SPECIFY;  ENGLISH  ( ) SPANISH!  ) TOTAL 


X $10.00. 


REGNO 


USD.  Inc.  - 2075-A  West  Park  Place,  Stone  Mountain,  Georgia  30087  (U.S.A) 
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Appendix  A.2  - Installing  EPI  INTO  5.0 


The  following  provides  instructions  for  installing  the  EPI  INFO  program  onto  either  your 
computer  hard  drive  or  onto  floppy  disks.  Chapter  4 of  the  EPI  INFO  Version  5 manual 
provides  more  detailed  instructions  for  installing  EPI  INFO  as  well  as  additional  information  on 
the  choices  available  at  different  points  of  the  installation  procedure. 


Installing  EPI  INFO  5,0  to  a Hard  Drive  (C:  or  D:) 

1.  Insert  Disk  1 of  EPI  INFO  into  drive  A (or  B,  depending  on  the  size  of  the  disk  and 
drive). 

2.  Type  A:  (or  B:),  and  press  the  ENTER  key. 

3.  Type  INSTALL,  and  press  the  ENTER  key. 

This  will  start  the  Install  program,  which  will  prompt  you  for  information  on  the  source 
drive  you  are  using,  the  destination  drive  you  want  to  install  EPI  INFO  on,  and  other 
information. 

4.  Enter  A (or  B)  for  the  "source  drive"  in  the  top  left  section  of  the  initial  Install  screen. 

5.  Enter  C (or  D)  for  the  "destination  drive"  in  the  top  right  section  of  the  screen. 

6.  Enter  N for  "no  removable  disks"  in  the  last  line  of  the  top  right  section  of  the  screen. 

7.  Enter  I for  "install  program  for  use"  in  the  line  under  top  two  sections.  (You  can  enter 
C at  a later  time  to  copy  the  EPI  INFO  Install  program  onto  disks  for  further  distribution 
of  the  EPI  INFO  program.  The  DOS  command  DISKCOPY  can  also  be  used  to  copy 
the  EPI  INFO  Installation  Disks). 

8.  Choices  for  the  video  card  that  is  used  by  your  system  are  indicated  in  the  bottom  right 
hand  section  of  the  screen.  Type  the  correct  number  in  bottom  left  section  (if  the 
computer  has  not  already  done  this),  and  press  the  ENTER  key. 

9.  Press  the  F4  key  to  continue  the  installation  procedure. 

10.  The  EPI  INFO  install  files  are  divided  into  groups  on  the  disks. 

On  the  "Installing  EPI  INFO"  screen  that  appears,  you  will  be  asked  to  indicate  the 
distribution  disk  number  with  which  the  installation  should  begin. 
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Select  1 to  install  the  basic  EPI  INFO  structure  (unless  you  have  previously  installed 
Group  1).  As  EPI  INFO  installs  itself,  it  will  "beep"  from  time  to  time  to  ask  you  which 
of  the  additional  eight  groups  you  want  to  have  installed. 

It  is  easiest  to  install  all  of  the  groups,  but  if  your  computer  has  limited  space,  do  not 
install  groups  6,  7 and  8. 

Press  the  ENTER  key  to  have  the  groups  you  want  installed.  Press  ESC  (the  Escape 
key)  to  skip  a group. 

Each  time  a group  is  chosen  for  installation,  the  screen  will  show  the  files  being 
installed. 

You  will  be  prompted  to  insert  additional  EPI  INFO  installation  disks  as  necessary. 

You  can  exit  the  Install  program  at  any  time  by  pressing  the  FIO  key  (unless  you  are 
copying  the  Install  program). 

11.  Following  the  installation  of  the  EPI  INFO  groups,  enter  Y to  edit  your 
AUTOEXEC.BAT  and  CONFIG.SYS  files. 

12.  EPI  INFO  is  now  installed.  Read  the  next  screen.  Enter  Y to  register  with  EPI  INFO, 
or  type  N to  end  the  installation  program. 


Starting  the  EPI  INFO  program  from  the  hard  (C:  or  D:)  drive 

• To  start  the  program,  enter  CD  EPI5  in  response  to  the  "C  prompt"  (the  "C:  \ > " prompt 
appears  at  the  left  of  the  screen). 

• When  C:\EPI5  appears  on  the  screen,  type  EPI  and  press  ENTER.  The  EPI  INFO 
Main  Menu  will  appear  on  the  screen. 


Setting  Printer  Options 

• If  you  have  a dot  matrix  printer,  you  will  not  have  to  worry  about  setting  any  printer 
options.  However,  if  you  have  a laser  printer  or  plotter  you  should  create  a special 
CONFIG. EPI  file  using  the  EPED  Text  Editor  program  listed  on  the  EPI  INFO  Main 
Menu. 


Choose  EPED  on  the  EPI  INFO  Main  Menu,  and  press  ENTER. 
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• If  you  have  a Hewlett  Packard  compatible  laser  printer,  type  in  the  following  on  the  "text 
screen"  that  appears: 

SET  PRINTER  = HP 
PLINES  = 60 

Press  the  F9  key  to  save  these  settings  into  a file.  When  prompted  for  the  file  name, 
type  CONFIG.EPI,  and  then  press  ENTER.  The  file  is  now  saved  and  will 
permanently  maintain  these  settings  (unless,  at  a later  date,  you  edit  the  CONFIG.EPI 
file  and  change  the  printer  settings). 

Press  the  FlO  key  to  exit  EPED. 

• If  you  have  a Hewlett  Packard  compatible  plotter,  refer  to  the  EPI  INFO  Version  5 
manual  for  the  printer  settings  required  for  plotters. 


Installing  EPI  INFO  onto  disks  (A:  and  B:) 

If  a "hard  drive"  is  not  available  on  the  computer  you  are  using,  make  sure  the  following  disks 
are  on  hand.  If  you  are  installing  to: 

• 5-1/4",  double-sided,  double  density  disks,  have  9 disks  on  hand; 

• 5-1/4",  double-sided,  high  density  disks,  3 disks  are  required; 

• 3-1/2",  double-sided,  double  density  disks,  5 disks  are  needed, 

• 3-1/2",  double-sided,  high  density  disks,  only  2 disks  are  needed. 

All  disks  should  be  formatted,  blank,  and  have  a blank  label. 

1.  Follow  Steps  1 through  4 provided  in  the  instructions  for  installing  EPI  INFO  to  a hard 
drive. 

2.  Enter  B (or  A)  for  the  "destination  drive"  in  the  top  right  section  of  the  screen. 

3.  Enter  Y for  "removable  disks"  in  the  last  line  of  the  top  right  section  of  the  screen. 

4.  Follow  Steps  7 through  10  provided  in  the  instructions  for  installing  EPI  INFO  to  a hard 
drive. 

Note  that: 

• one  installed  group  will  require  one  5-1/4"  double  sided,  double  density  disk; 
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• two  to  three  installed  groups  will  fit  on  one  5-1/4"  double  sided,  high  density 
disk, 

• two  installed  groups  will  fit  on  one  3-1/2"  double  sided,  double  density  disk, 

• four  to  five  installed  groups  will  fit  on  one  3-1/2"  double  sided,  high  density  disk. 

Change  disks  as  prompted,  then  label  each  disk  with  the  name  of  the  groups  that  have 
been  installed  on  it. 

5.  Follow  Steps  11  and  12  provided  in  the  instructions  for  installing  EPI  INFO  to  a hard 
drive. 


Starting  the  EPI  INFO  program  from  a disk  (A:  or  B:)  drive 

• Insert  Disk  1 of  the  installed  EPI  INFO  program  in  the  floppy  drive. 

• At  the  A:  (or  B:)  prompt,  type  EPI  and  press  the  ENTER  key.  The  EPI  INFO  Main 
Menu  will  appear  on  the  screen. 


Setting  Printer  Options 

• If  you  have  a dot  matrix  printer,  you  will  not  have  to  worry  about  setting  any  printer 
options.  However,  if  you  have  a laser  printer  or  plotter  you  should  create  a special 
CONFIG. EPI  file  using  the  EPED  program  listed  on  the  EPI  INFO  Main  Menu. 

Follow  instructions  provided  in  the  section  on  installing  EPI  INFO  to  a hard  drive  for 
creating  a printer  setup  file. 
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Appendix  A.3  - Data  Entry  and  Analysis  Using  EPI  INTO  5.0 


The  3 basic  programs  in  EPI  INFO  5.0  include: 

• EPED,  a Text  Editor  program  that  can  be  used  for  general  word  processing  or  creating 
other  text  files,  including  questionnaires; 

• ENTER,  a Data  Entry  program  that  is  used  to  enter  responses  to  a questionnaire  or  other 
data  into  a database;  and 

• ANALYSIS,  a Data  Analysis  program.  ANALYSIS  can  be  used  to  produce  a list  of  data 
records,  to  manipulate  data,  to  calculate  basic  descriptive  statistics  (including  frequencies 
and  means)  and  other  inferential  statistical  analyses,  and  to  produce  graphs. 

Other  more  advanced  programs  that  are  useful  for  data  management  are  available  in 
EPI  INFO  5.0: 

• CHECK  is  a program  that  can  detect  errors  in  data  entry,  by  checking  coding  of  values, 
legal  values,  skip  patterns,  etc.; 

• CONVERT  converts  data  files  from  an  EPI  INFO  format  into  other  standard  formats 
(e.g.,  SAS,  SPSS-X,  LOTUS  1-2-3,  dBASE  II  and  III,  and  Statpac); 

• IMPORT  transforms  files  created  in  other  software  packages  for  use  in  EPI  INFO;  and 

• MERGE  is  used  to  combine  separate  data  files  with  the  same  structure  into  a single 
database. 

There  are  two  programs  you  will  probably  rarely  use: 

• STATCALC  which  calculates  statistics  for  2 by  2 to  2 by  9 tables  (this  can  also  be  done 
using  the  ANALYSIS  program);  and 

• VALIDATE  which  can  be  used  to  compare  files  entered  by  two  different  people  to 
identify  discrepancies  in  data  entry. 

The  following  pages  provide  a step-by-step  guide  that  describes  the  process  of  preparing  data 
for  analysis  and  conducting  some  simple  statistical  analyses  using  the  EPED,  ENTER,  and 
ANALYSIS  programs  in  EPI  INFO  5.0.  (You  can  explore  these  programs  on  your  own,  or  you 
can  refer  to  the  EPI  INFO  Version  5.0  manual  for  more  information  on  the  programs  available 
in  EPI  INFO.) 
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The  hypothetical  data  used  in  this  example  is  taken  from  the  fictional  prevention  program  for 
high-risk  youth  described  in  Example  H from  the  casebook. 

You  will  recall  that  clients  who  attended  the  Youth  Addiction  Services  Education  Program 
completed  a feedback  form  at  the  end  of  the  fourth  session  of  the  program  (see  Section  H,  Part 
2).  These  questionnaires  were  completed  anonymously.  At  the  end  of  the  year  the  program 
secretary  gathered  all  the  feedback  questionnaires  that  clients  had  completed  and  identified  the 
information  on  the  forms  that  would  be  used  for  the  analyses.  Although  the  instructions  that 
follow  in  this  section  do  not  closely  follow  the  procedures  that  were  used  to  generate  the 
statistical  output  included  in  Section  H,  Part  2 and  3,  they  will  show  how  data  collected  on 
questionnaires  can  be  prepared  for  statistical  analysis  and  then  be  analyzed. 


Preparing  the  Questionnaire  for  Data  Entry 

1.  Using  a blank  Feedback  Questionnaire,  circle  each  "variable,"  or  piece  of  information 
that  you  are  interested  in  analyzing. 

Because  the  forms  were  completed  anonymously,  you  should  also  add  the  variable  "ID" 
to  the  top  right  hand  comer  of  the  form.  Each  questionnaire  should  be  given  a unique 
number  to  identify  the  questionnaire.  These  identification  numbers  are  not  intended  to 
be  Client  ID^’s  that  identify  actual  clients,  but  are  used  simply  to  identify  a questionnaire 
for  cross-checking  purposes  if  questions  arise  at  a later  date  about  the  integrity  of  the 
data  that  was  entered  into  the  computer. 

2.  Give  each  variable  a name  that  consists  of  10  characters  or  less  (e.g.,  "Identification"  is 
too  long  - it  is  14  characters  long;  "ID"  is  acceptable  at  two  characters).  In  this  example 
we  use  ID,  ALCOHOL,  DRUGS,  WHYUSE,  HEALTHIER,  VIDEOS,  DISCUSSION, 
HANDOUTS,  AGE,  GENDER,  and  REFERRAL  as  variable  names. 

3.  Next,  identify  the  variable  type  (and  format  when  appropriate)  for  each  variable.  This 
information  will  ensure  consistency  in  data  coding  and  data  entry.  In  this  example, 
variable  types  include: 

(a)  sequential  identification  number:  <idnum>*; 

(b)  numeric  variable:  #,  ##.#,  etc.,  which  indicates  the  number  of  digits,  and 
whether  the  number  includes  a decimal  point;  and 

(c)  alphanumeric  variable  (letters  and/or  numbers):  < A > , < A > which  indicates 
the  number  of  characters  used  for  coding  a variable. 

1*  It  should  be  noted  that  when  actual  Client  ID  numbers  are  used,  a numeric  variable 
type  should  be  used.  This  would  give  you  control  over  the  ID  numbers  that  would  be 
entered  for  individual  clients  or  respondents.] 
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More  information  on  variable  field  types  is  provided  in  Chapter  7 of  the  EPI  INFO  5.0 
Manual;  also,  the  sub-menu  "Questions”  under  the  Text  Menu  in  the  EPED  Text  Editor 
program  (which  is  accessed  by  pressing  the  F4  key)  lists  all  field  types  used  by  the  EPI 
INFO  statistical  program. 

Figure  1 shows  the  results  of  following  Steps  1-3. 

4.  At  this  point,  going  back  to  each  feedback  questionnaire  that  was  completed  by  program 
participants  and  highlighting  the  information  on  the  forms  that  will  be  entered  on  the 
computer  would  make  the  task  of  entering  the  data  easier. 

But  before  you  can  enter  the  data  you  have  to  create  a data  entry  form  using  the  EPED 
Text  Editor  and  ENTER  Data  Entry  programs  in  EPI  INFO  5.0. 


Figure  1. 
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Youth  Addiction  Services  Education  Program 

Feedback  Questionnaire 


ID  <idnum> 


Please  rate  how  useful  you  found  each  of  the  four  education  sessions: 

Check  here 

Circle  one  if  you  did 


not  at  all 
useful 

very 

useful 

not  attend 
that  session 

1. 

Alcohol  and  its  effects 

1 

2 

3 

4 

5 

□ 

ALCOHOL  ft 

11. 

Other  drugs  and  their  effects 

1 

2 

3 

4 

5 

□ 

DRUGS  # 

III. 

Why  people  use 

1 

2 

3 

4 

5 

□ 

WHYUSE  it 

IV. 

Making  healthier  choices 

1 

2 

3 

4 

5 

□ 

HEALTHIER  # 

Please  rate  how  useful  you  found  each  of  the  three  different  presentation  formats. 

Circle  one 

not  at  all  very 

useful  useful 


Videos 

1 

2 

3 4 5 

VIDEOS  a 

Group  discussion 

1 

2 

3 4 5 

DISCUSSION  a 

Handouts 

1 

2 

3 4 5 

HANDOUTS  a 

Age: 

AGE  an 

Gender: 

□ 

male 

□ female 

GENDER  < A > 

Who  referred  you  to  the  program? 

(Check  one  only 

- if  uncertain,  ask  counsellor.) 

1.  Q family  or  friends 

2.  □ legal  system 

3.  □ school  REFERRAL  # 

4.  □ self 

5.  □ other 


THANK  YOU  FOR  HELPING  US  TO  IMPROVE  OUR  PROGRAM  ! 
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Creating  a "Data  Entry  Form  " using  the  EPED  Text  Editor 


NOTES  ON  USING  EPI  INFO 

To  move  through  a text  or  screen  use  the  arrow  keys  (t  I ♦-  ->). 

To  exit  or  quit  any  program  in  EPI  INFO,  press  FIO. 

To  return  to  the  Main  Menu  at  any  time,  press  ESC  until  the  Main  Menu  appears. 


EPED  is  a Text  Editor  program  that  can  be  used  for  general  word  processing  or  creating  other 
text  files,  including  questionnaires. 

1 . Start  the  EPI  INFO  5.0  program  by  entering  EPI  in  response  to  the  C:\EPI5  > prompt, 
or  A:  > prompt  (see  instructions  on  starting  the  EPI  INFO  program  under  "Installing 
EPI  INFO  to  a Hard  Drive",  and  "Installing  EPI  INFO  onto  disks"). 

The  EPI  INFO  Main  Menu  will  appear  on  the  screen. 

2.  Choose  EPED  on  the  EPI  INFO  Main  Menu. 

3.  A "text  screen"  will  appear  (a  screen  with  a "menu"  across  the  top,  some  information 
across  the  bottom,  and  a blank  section  in  the  middle).  You  will  type  all  of  your 
variable  names  and  indicate  variable  types  and  formats  on  this  screen  and  then  save  the 
file  as  a questionnaire  file. 

This  questionnaire  text  file  will  later  be  used  as  a "data  entry  form"  (a  programmed 
form  which  makes  it  easy  to  enter  data  into  a data  file). 

If  you  are  preparing  a questionnaire  or  other  data  collection  forms  at  the  beginning  of 
an  evaluation  project,  you  can  use  EPED  to  create  the  forms.  Even  if  the  questionnaire 
has  already  been  prepared  (as  is  the  case  in  this  example),  a questionnaire  text  file  must 
be  created. 

Simply  type  in  the  questions  that  appear  on  the  questionnaire,  or  the  variable  names  you 
have  selected  for  your  data. 

Using  the  example  presented  above  in  "Preparing  the  Data  for  Analyses,"  type  ID  on 
the  first  line  of  the  text  screen,  then  press  the  F4  key  for  the  TEXT  menu,  and  choose 
Questions.  Next,  choose  the  <idnum>  format.  Type  5 to  indicate  the  number  of 
digits  to  use  for  this  field. 
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On  the  screen  you  will  see: 

ID  <idnum> 

Press  the  ENTER  key. 

4.  Next,  type  ALCOHOL.  Press  the  F4  key,  choose  Questions,  and  select  ##.  Type  1 
for  the  number  of  digits  before  the  decimal,  and  press  ENTER.  Type  0 for  the  number 
of  digits  after  the  decimal,  and  press  ENTER. 

On  the  screen  you  will  see: 

ALCOHOL  # 

Press  the  ENTER  key. 

5.  Repeat  Step  4 for  each  of  the  remaining  variables,  making  the  following  substitutions: 


Variable  Name 

Variable  Type  and  Format* 

DRUGS 

#(1,0  decimals) 

WHYUSE 

#(1,0  decimals) 

HEALTHIER 

#(1,0  decimals) 

VIDEOS 

#(1,0  decimals) 

DISCUSSION 

#(1,0  decimals) 

HANDOUTS 

#(1,0  decimals) 

AGE 

##  (2,  0 decimals) 

GENDER 

<A>  (1) 

REFERRAL 

#(1,0  decimals) 

[*  EPI  INFO  will  prompt  you  for  the  number  of  digits  or  characters  for  idnum, 
text,  numeric,  and  alphanumeric  data.  The  numbers  you  should  enter  for  the 
variables  listed  in  the  table  are  indicated  in  brackets.] 

Figure  2 displays  what  you  should  see  on  your  screen  when  you  have  finished  typing 
in  all  of  the  variable  fields  for  your  Feedback  Questionnaire  form). 
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Figure  2.  Feedback  Questionnaire  (FEEDBACK.QES) 


ID  <idnum> 
ALCOHOL  # 
DRUGS  # 
WHYUSE  # 
HEALTHIER  # 
VIDEOS  # 
DISCUSSION  # 
HANDOUTS  # 
AGE  m 

GENDER  <A> 
REFERRAL  # 


6.  Check  your  entries  for  accuracy.  You  can  use  the  arrow  keys  (t  I to  move 

around  the  screen.  If  you  have  made  a mistake,  simply  delete  your  entry  using  the 
DEL  key  or  BACKSPACE  key,  and  re-enter  the  variable  name  and  format. 

Press  the  F9  key  to  save  your  file.  You  will  need  to  provide  a filename  to  save  your 
file.  Filenames  cannot  be  any  longer  than  eight  letters,  but  can  be  followed  by  a period 
and  up  to  three  letters  can  be  used  for  an  extension. 

File  extensions  are  often  used  to  identify  the  contents  of  the  file.  EPI  INFO  uses  three 
main  extensions  to  identify  the  following  types  of  files: 

• .QES  for  an  EPI  INFO  questionnaire  file 

• .REC  for  an  EPI  INFO  data  or  record  file 

• .CHK  for  an  EPI  INFO  check  commands  file 

Type  FEEDBACK.QES  and  press  ENTER. 

The  "extension  .QES"  identifies  the  file  as  an  EPI  INFO  questionnaire  file. 

The  EPI  INFO  questionnaire  file  will  be  converted  into  a "data  entry  form"  in  the  next 
section  which  describes  the  EPI  INFO  ENTER  or  data  entry  program. 

7.  Press  FIO  key  to  exit  the  EPED  Text  Editor  program. 
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Description  of  Some  EPED  Functions  and  Other  Helpful  Hints 
1)  Function  keys  in  EPED: 

Ten  "function"  keys  are  used  in  EPED.  To  activate  a function,  simply  press  the  identified  key. 
To  exit  from  any  of  the  functions,  press  ESC  (the  Escape  key).  We  would  like  to  draw 
particular  attention  to  the  F3  key  which  brings  up  the  EPIAH)  tutorial  for  EPED. 


• FI  - Hip: 

provides  help  on  different  topics  related  to  the  EPED  program 

• F2  - File: 

lists  file-related  functions  available  in  EPED 

• F3  - EPIAID: 

accesses  the  EPED  tutorial 

• F4  - Txt: 

includes  a number  of  text  related  functions 

• F5  - Print: 

activates  the  print  setup  screen 

• F6  - Set: 

lists  setup  functions 

• F7  - Find: 

lists  a number  of  find,  replace  and  move  functions 

• F8  - Blk: 

allows  you  to  perform  copy,  move,  delete  and  other  functions  on  an 
identified  block  of  text 

• F9-Save: 

saves  your  file 

• FIO  - Done: 

exits  EPED  and  returns  to  the  EPI  INFO  main  menu 

2)  Helpful  Hints  on  Using  EPED  to  Create  a Questionnaire 

• NEVER  include  the  following  characters  in  the  text  of  the  questionnaire  as  they  are  data 
entry  field  identifiers; 

(underscore) 

# (number  sign) 

< or  > (less  than  or  greater  than  signs) 
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• Variable  names  cannot  be  longer  than  10  characters.  If  a question  on  the  questionnaire  is 
longer  than  10  characters,  you  can  place  { }’s  around  the  characters  you  want  to  be  used  for 
the  variable  name  on  the  data  entry  form.  For  example: 


What  appears  on  the  questionnaire? 

Variable  Name 

Age 

AGE 

What  is  your  age  in  years? 

WHATISYOUR 

What  is  your  {age}  in  years? 

AGE 

What  is  your  {age}  in  {years}? 

AGEYEARS 

What  is  your  {age  in  years}? 

AGEINYEARS 
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Entering  Data  using  the  ENTER  Data  Entry  Program 

NOTES  ON  USING  EPI  INFO 

• To  move  throujgh  a text  or  screen  iise  the  arrow  keys  ( t A >-  -♦), 

• To  exit  or  quit  any  program  in  EPI  INFO,  press  FIO. 

• To  return  to  the  Main  Mrau  at  any  time,  press  ESC  until  the  Main  Menu  appears 


Once  the  questionnaire  is  completed,  data  may  be  entered  into  a computer  file  using  ENTER, 
the  EPI  INFO  Data  Entry  program  that  is  used  to  enter  responses  to  a questionnaire  or  other 
data  into  a database. 

1.  Highlight  ENTER  on  the  EPI  INFO  Main  Menu,  and  press  ENTER. 

2.  You  will  be  asked  for  the  name  of  the  data  or  records  file  you  want  to  use  to  enter  data. 

Because  the  "record  file"  for  FEEDBACK. QES  does  not  yet  exist,  you  need  to  create  it  at 
this  point. 

Type  in  FEEDBACK  to  identify  the  record  file  you  want  to  create  as  FEEDBACK.REC. 
Press  ENTER. 

EPI  INFO  will  respond  with  a note  that  FEEDBACK.REC  will  be  created. 

Press  ESC  (the  Escape  key)  to  create  FEEDBACK.REC. 

3.  EPI  INFO  will  ask  you  to  specify  the  questionnaire  file  you  want  to  base  your  record  file  on. 

Type  in  FEEDBACK  to  identify  the  associated  questionnaire  file  as  FEEDBACK.QES. 

EPI  INFO  will  create  a record  file  from  FEEDBACK.QES,  and  a data  entry  form  that  is 
similar  to  Figure  2 will  appear  on  the  screen.  The  highlighted  areas  beside  each  listed 
"variable  name"  are  called  "data  entry  fields."  You  will  enter  data  in  these  fields. 

Steps  4 through  9 describe  EPI  INFO  procedures  for  entering  hypothetical  data  for  the  Youth 
Addiction  Services  Education  Program  Feedback  Questionnaire.  The  data  that  are  used  in  this 
section  and  subsequent  sections  correspond  to  the  data  that  were  used  in  Section  H.  If  you  like, 
you  can  enter  a set  of  your  own  fictitious  data,  or  you  can  use  the  data  we  have  used  here. 


258 


Figure  3 provides  a listing  of  the  data  set  used  in  Section  H (Figure  3 is  presented  in  the  next 
section  on  "Analyzing  Data  using  the  ANALYSIS  program"). 

4.  The  Client  ID  number  will  appear  automatically.  Ensure  that  the  ID  number  that  appears  on 
the  screen  corresponds  to  the  ID  number  you  record  on  the  questionnaire. 

The  cursor  will  move  to  the  beginning  of  the  next  field. 

5.  Type  in  responses  in  the  appropriate  range  for  all  of  the  variables  (e.g.,  ALCOHOL:  1-5; 
GENDER:  M,F),  or  enter  the  data  listed  in  Figure  3 for  Rec  1. 

Just  above  the  bottom  line  of  the  screen  at  the  left  both  the  name  of  the  current  variable  and 
the  type  of  responses  that  will  be  accepted  for  the  variable  are  identified. 

In  the  case  of  missing  data,  press  ENTER  to  skip  the  field. 

6.  After  data  has  been  entered  in  the  last  field  of  the  screen,  the  following  appears  in  place  of 
the  variable  name  at  the  bottom  of  the  screen: 

Write  data  to  disk  (Y/N)? 

Check  to  ensure  the  accuracy  of  the  data  entered  (this  is  an  especially  important  step  when 
entering  real  data).  If  corrections  need  to  be  made,  type  N to  return  to  the  first  data  entry 
field.  Use  the  arrow  keys  (t  4)  to  move  to  the  field(s)  that  requires  correction  and  make 
the  necessary  modification(s). 

When  all  corrections  have  been  made,  press  the  END  key  to  move  to  the  bottom  of  the 
record.  You  will  be  asked  if  you  want  to  write  the  data  to  disk.  Type  Y to  add  the  record 
to  the  database. 

In  the  bottom  right  hand  comer,  "Rec=  1"  will  change  to  "Rec=  2".  The  Record  number 
increases  by  one  each  time  a new  record  is  completed  and  saved. 

7.  Repeat  Steps  4 through  6 to  enter  the  data  for  all  of  the  cases. 

8.  When  all  "cases"  or  "records"  have  been  entered  and  written  to  disk,  press  FIO  to  exit  from 
the  ENTER  program. 


Description  of  Some  ENTER  Functions  and  Other  Helpful  Hints 
1)  Function  keys  in  ENTER: 

Across  the  bottom  of  the  screen,  the  following  function  keys  used  in  ENTER  are  listed: 


259 


• < CTRL-N  > - New: 

use  to  enter  a new  record; 

• <CTRL-r>  - Find: 

will  brings  up  a sub-menu  that  lists  the  special  function  keys  used 
to  find  records; 

• F5  - Print: 

will  print  data  entries  for  all  records  or  a specified  subset  of  the 
records; 

• F6  - Delete: 

will  mark  the  record  as  deleted  with  an  asterisk  (*);  deleted 
records  are  ignored  when  conducting  analyses; 

• F9  - Choices: 

shows  any  codes  that  have  been  ignored  in  the  setup  for  automatic 
coding  in  a specified  field; 

• FIO  - Done: 

exits  ENTER  and  returns  to  the  EPI  INFO  main  menu. 

2)  Helpful  Hints  on  Using  ENTER  to  Enter  Data: 

• When  entering  or  editing  data,  remember  that  pressing  END  will  move  the  cursor  to  the  end 
of  the  record  and  bring  up  the  "Write  data  to  disk?"  prompt. 
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Analyzing  Data  using  the  ANALYSIS  Program 


NOTES  ON  USING  EPI  INFO 

• To  move  through  a text  or  screen  use  the  arrow  keys  ( t i **-  -♦). 

• To  exit  or  quit  any  program  in  EPI  INFO,  press  FIO. 

• To  return  to  the  Main  Menu  at  any  time,  press  ESC  until  the  Main  Menu  appears. 


ANALYSIS  is  a Data  Analysis  program  that  can  be  used  to  produce  a list  of  data  records, 
manipulate  data,  calculate  basic  descriptive  statistics  (including  frequencies  and  means)  and  other 
inferential  statistical  analyses,  and  produce  graphs.  The  bottom  section  of  the  screen  displays 
the  commands  you  select  from  sub-menus  in  ANALYSIS.  You  can  also  type  in  commands 
yourself  in  this  section  of  the  screen.  The  following  instructions  are  based  on  use  of  the  sub- 
menus for  developing  and  executing  command  statements. 

Selecting  a Data  File  and  Viewing  the  Database 

1.  Highlight  ANALYSIS  on  the  EPI  INFO  Main  Menu,  and  press  ENTER. 

2.  Type  READ  FEEDBACK.REC  to  select  the  Feedback  record  file  for  analysis.  Press 
ENTER. 

3.  Press  the  F2  key  for  the  Command  Menu,  choose  BROWSE  and  press  ENTER.  Press 
ENTER  again.  The  database  will  appear  on  the  screen.  Use  the  arrow  keys  (t  4 

to  move  through  the  database. 

4.  Press  ESC  (the  Escape  key)  to  return  to  the  ANALYSIS  screen. 

Accessing  the  Printer  and  Creating  a Heading  for  Printer  Output 

The  output,  or  results  of  every  command  that  is  executed  in  ANALYSIS,  can  be  printed  to  three 
output  formats  - the  screen,  a printer,  or  a file.  The  default  is  set  to  show  all  output  on  the 
screen. 

• To  print  to  the  printer  you  can: 

1)  press  F5  until  the  bottom  of  screen  reads  "F5  - PRINTER  OFF"  iyesi  to  print 
to  the  printer,  you  turn  it  to  offlll)\  or 


2)  press  F2,  highlight  ROUTE,  press  ENTER,  type  PRINTER  and  press 
ENTER;  or 

3)  type  ROUTE  PRINTER  and  press  ENTER. 

• To  print  to  a file  (that  can  be  edited  or  printed  later): 

1)  press  F2,  highlight  ROUTE,  press  ENTER,  type  in  a filename  and  press 
ENTER;  or 

2)  type  ROUTE  (filename. ext)  and  press  ENTER. 

Examples  of  extensions  you  can  use  to  identify  the  output  file  include  .OUT  (for  "output"), 
.LIS  (for  "listing"),  .RES  (for  "results"),  .SUM  (for  "summary"),  or  .DOC  (for 
"document").  So,  for  example,  the  filename  FB-FREQS.OUT  could  be  used  to  identify  a 
file  that  contains  the  output  of  frequency  analyses  conducted  on  FEEDBACK. REC  data. 

If  you  want  to  direct  the  output  to  a file  on  a floppy  disk,  remember  to  identify  the  disk 
when  specifying  the  filename  (e.g.,  A:\FB-FREQS.OUT). 

The  following  provides  instructions  for  directing  output  to  your  printer  and  specifying  a header 
for  your  output: 

1 . Press  the  F5  key  so  that  the  printer  status  indicator  on  the  bottom  of  the  screen  switches  to 
"FS-Printer  off."  Your  output  will  now  automatically  be  sent  to  the  printer  for  the  duration 
of  your  analysis  session. 

2.  Press  F2.  Highlight  HEADER,  and  press  ENTER.  Type  "1_"  (the  number  1 and  then  one 
space)  and  your  name.  Press  ENTER. 

Your  name  will  now  be  printed  at  the  top  of  each  page  of  output  that  is  printed  (that  is, 
unless  you  decide  to  change  the  header  for  different  runs  during  the  analysis  session). 

Checking  the  Data  (to  make  sure  that  they  are  accurate) 

1.  Press  F2.  Highlight  LIST,  type  *,  and  press  ENTER. 

A listing  of  the  data  will  be  shown  on  the  screen  of  your  computer  (see  Figure  3).  The 
listing  will  also  be  sent  to  your  printer. 

The  listing  of  the  data  that  were  used  in  Section  H in  the  casebook  is  shown  in  Figure  3. 

2.  If  you  entered  the  data  from  Section  H,  check  the  accuracy  of  your  data  against  the  data 
listed  in  Figure  3.  When  you  are  entering  real  data,  you  would  check  your  data  listing 
against  the  responses  made  in  the  original  questionnaires. 
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Figure  3.  LIST*  - List  of  all  variables  of  all  records 


REC 

ID 

ALCOHOL 

DRUGS 

WHYUSE 

HEALTHIER 

VIDEOS 

DISCUSSION 

HANDOUTS 

AGE  GENDER 

REFERRAL 

1 

1 

4 

4 

4 

4 

4 

4 

3 

16  F 

2 

2 

2 

3 

4 

4 

4 

3 

4 

3 

17  M 

3 

3 

3 

3 

3 

5 

5 

3 

4 

4 

18  F 

2 

4 

4 

1 

, 

• 

, 

16  M 

3 

5 

5 

5 

5 

5 

5 

5 

5 

5 

16  F 

3 

6 

6 

3 

4 

4 

4 

4 

4 

4 

19  M 

1 

7 

7 

5 

5 

5 

5 

5 

5 

5 

16  F 

3 

8 

8 

5 

5 

5 

5 

4 

5 

5 

20  M 

5 

9 

9 

4 

4 

3 

3 

3 

5 

4 

17  M 

2 

10 

10 

5 

5 

5 

5 

5 

5 

5 

16  F 

1 

11 

11 

2 

, 

16  F 

3 

12 

12 

3 

3 

3 

4 

4 

5 

2 

18  F 

2 

13 

13 

2 

3 

3 

4 

2 

5 

2 

19  M 

3 

14 

14 

5 

5 

5 

5 

5 

5 

5 

16  F 

1 

15 

15 

3 

5 

4 

3 

2 

3 

2 

17  M 

2 

16 

16 

3 

3 

16  F 

1 

17 

17 

3 

4 

3 

4 

3 

4 

3 

19  F 

2 

18 

18 

5 

5 

5 

5 

5 

4 

4 

16  M 

3 

19 

19 

5 

5 

5 

5 

5 

5 

3 

17  F 

3 

20 

20 

2 

, 

16  F 

3 

21 

21 

3 

5 

5 

5 

4 

5 

5 

16  F 

4 

22 

22 

2 

2 

2 

2 

3 

4 

2 

21  M 

2 

23 

23 

5 

4 

4 

4 

5 

5 

4 

19  F 

4 

24 

24 

5 

5 

5 

5 

5 

5 

5 

16  F 

5 

25 

25 

2 

2 

2 

2 

2 

2 

2 

21  M 

2 

26 

26 

4 

4 

4 

5 

3 

5 

4 

19  M 

1 

27 

27 

3 

3 

3 

4 

3 

4 

4 

23  M 

4 

28 

28 

3 

3 

3 

4 

3 

4 

3 

18  M 

2 

29 

29 

5 

5 

5 

5 

5 

5 

5 

16  F 

1 

30 

30 

4 

5 

5 

5 

5 

5 

4 

16  M 

4 

31 

31 

4 

21  M 

1 

32 

32 

5 

5 

4 

5 

5 

5 

4 

16  F 

3 

33 

33 

5 

5 

5 

5 

5 

5 

5 

16  F 

5 

34 

34 

5 

5 

5 

5 

5 

5 

5 

19  M 

5 

35 

35 

5 

4 

5 

5 

5 

5 

5 

17  F 

3 

36 

36 

4 

3 

3 

3 

3 

5 

4 

16  M 

2 

37 

37 

2 

2 

2 

2 

2 

3 

2 

19  M 

2 

38 

38 

, 

16  M 

2 

39 

39 

3 

4 

4 

4 

3 

5 

5 

18  F 

4 

40 

40 

4 

5 

5 

5 

4 

4 

4 

16  M 

4 

41 

41 

4 

4 

5 

5 

3 

4 

4 

21  M 

5 

42 

42 

4 

5 

5 

5 

4 

4 

4 

16  F 

4 

43 

43 

4 

4 

, 

, 

17  M 

3 

44 

44 

3 

4 

4 

4 

3 

3 

3 

18  M 

3 

45 

45 

5 

5 

5 

5 

5 

5 

4 

17  M 

3 

46 

46 

4 

5 

5 

5 

4 

5 

4 

16  M 

1 
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If  there  are  corrections  to  be  made,  circle  the  mistakes  on  the  printout. 

Press  F2;  choose  UPDATE  and  press  ENTER.  The  database  will  appear  on  the  screen. 

In  the  top  left  hand  comer,  the  current  data  entry  field  is  identified.  The  Record  number 
appears  in  the  left  hand  column.  Use  the  arrow  keys  ( T I «-  -»)  to  move  to  the  information 
you  circled  on  the  printout. 

Type  in  the  correct  data,  and  press  ENTER. 

When  all  corrections  are  made,  press  ESC  (the  Escape  key). 

3.  The  database  should  now  be  correct.  Check  the  database  again  by  printing  a listing  of  the 
data  and  checking  it  against:  1)  the  corrections  that  were  indicated,  or  2)  against  the  original 
questionnaires  to  thoroughly  verify  the  accuracy  of  the  database. 

Press  F2.  Highlight  NEWPAGE,  and  press  ENTER. 

Press  F2.  Highlight  LIST,  type  *,  and  press  ENTER. 

A second  listing  of  the  database  will  be  printed. 

Once  it  is  confirmed  that  the  data  are  accurate,  continue  with  the  statistical  analyses 
presented  in  the  following  examples. 

Examples  of  Statistical  Analyses 

#1-  To  generate  FREQuencies  of  categorical  responses  (GENDER,  REFERRAL): 

1.  Press  F2.  Highlight  NEWPAGE,  and  press  ENTER.  Press  ENTER  a second  time. 

2.  Press  F2.  Highlight  FREQ,  and  press  ENTER. 

3.  Press  F3.  Highlight  GENDER,  and  press  ENTER. 

4.  Repeat  Step  2,  choosing  REFERRAL  as  the  next  categorial  variable. 

5.  The  bottom  line  of  the  screen,  or  the  command  line,  should  read: 

FREQ  GENDER  REFERRAL 
Press  ENTER. 

Frequency  tables  for  each  of  the  three  variables  will  be  generated  at  the  printer. 
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6. 

See  Figure  4 which  shows  the  output  from  this  analysis 

Figure  4.  Output  from  FREQ  GENDER  REFERRAL 

GENDER  1 

Freq  Percent 

Cum. 

1 ! 

25  54.3% 

54.3% 

2 1 

21  45.7% 

100.0% 

Total  j 

46  100.0% 

Sum 

= 

67.00 

Mean 

= 

1.46 

Standard 

deviation  = 

0.50 

REFERRAL 

} Freq  Percent 

Cum. 

1 

1 8 17.4% 

17.4% 

2 

1 12  26.1% 

43.5% 

3 

1 14  30.4% 

73.9% 

4 

1 7 15.2% 

89.1% 

5 

1 5 10.9% 

100.0% 

Total  J 46  100.0% 


Sum  = 127.00 
Mean  = 2.76 
Standard  deviation  = 1.23 


7.  Continue  to  ^2,  or  press  FlO  to  quit  ANALYSES.  At  this  point  you  can  choose  QUIT 
on  the  EPI  INFO  Main  Menu  to  end  the  session. 

#2-  To  generate  MEANS  or  averages  of  continuous  variables  (AGE,  ALCOHOL,  VIDEOS); 

(Note:  MEANS  analyses  must  be  conducted  separately  for  each  variable.) 

1.  Press  F2.  Choose  NEWPAGE,  and  press  ENTER. 

2.  Press  F2.  Highlight  MEANS,  and  press  ENTER. 

3.  Press  F3.  Highlight  AGE,  and  press  ENTER.  Press  ENTER  again. 

A printout  of  the  results  of  the  MEANS  analysis  will  be  generated  at  the  printer. 

4.  Repeat  steps  1 through  3,  choosing  the  variable  ALCOHOL  in  Step  2. 

5.  Repeat  steps  1 through  3,  choosing  the  variable  VIDEOS  in  Step  2. 

6.  See  Figure  5 which  shows  the  output  from  these  analyses. 
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7.  Continue  to  ^3,  or  press  FIO  to  quit  ANALYSES.  At  this  point  you  can  choose  QUIT 
on  the  EPI  INFO  Main  Menu  to  end  the  session. 

#3-  To  generate  TABLES  or  Crosstabs  for  two  categorical  variables  (GENDER, 
REFERRAL): 

1.  Press  F2.  Highlight  NEWPAGE,  and  press  ENTER. 

2.  Press  F2.  Highlight  TABLES,  and  press  ENTER. 

3.  Press  F3  and  choose  GENDER. 

4.  Press  F3  and  choose  REFERRAL. 

5.  The  command  line  should  read: 

TABLES  GENDER  REFERRAL 
Press  ENTER  to  generate  the  crosstabs  table. 

The  crosstabs  table  will  be  generated  at  the  printer. 

6.  See  Figure  6 which  shows  the  output  from  this  analysis. 

7.  Continue  to  #4,  or  press  FIO  to  quit  ANALYSES.  At  this  point  you  can  choose  QUIT 
on  the  EPI  INFO  Main  Menu  to  end  the  session. 

#4-  To  generate  MEANS  or  averages  of  AGE  for  each  category  of  GENDER: 

1.  Press  F2.  Highlight  MEANS  and  press  ENTER. 

2.  Press  F3  and  choose  AGE. 

3.  Press  F3  and  choose  GENDER. 

4.  The  command  line  should  now  read: 

MEANS  AGE  GENDER 

Press  ENTER  to  generate  the  ANOVA  table  and  statistics. 

A printout  of  the  results  of  the  MEANS  analysis  will  be  generated  at  the  printer. 

5.  See  Figure  7 which  shows  the  output  from  this  analysis. 
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6.  Press  FIO  to  quit  ANALYSES.  At  this  point  you  can  choose  QUIT  on  the  EPI  INFO 
Main  Menu  to  end  the  session. 


Figure  5.  Output  from  MEANS  AGE;  MEANS  ALCOHOL;  MEANS  VIDEOS 


AGE 


Freq 


Percent 

Cum. 

45.7% 

45.7% 

15.2% 

60.9% 

10.9% 

71.7% 

15.2% 

87.0% 

2.2% 

89.1% 

8.7% 

97.8% 

2.2% 

100.0% 

100.0% 

805.00 

= 

17.50 

ion  = 

1.83 

16 

17 

18 

19 

20 
21 
23 


21 

7 

5 

7 

1 

4 

1 


Total 

Sum 

Mean 


ALCOHOL 


Freq  Percent  Cum. 


1 

6 

12 

11 

15 


2.2% 

13.3% 

26.7% 

24.4% 

33.3% 


2.2% 

15.6% 

42.2% 

66.7% 

100.0% 


Total 


45  100.0% 


Sum  = 168.00 
Mean  = 3.73 
Standard  deviation  = 1.14 


VIDEOS  1 

Freq 

Percent 

Cum. 

2 1 

4 

10.3% 

10.3% 

3 1 

12 

30.8% 

41.0% 

4 1 

8 

20.5% 

61.5% 

5 1 

15 

38.5% 

100.0% 

Total  I 

Sum 

Mean 

Standard 

39  100.0% 

deviation  = 

151.00 

3.87 

1.06 
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Description  of  Some  ANALYSIS  Functions  and  Other  Helpful  Hints 
1)  Function  keys  in  ANALYSIS: 

Seven  function  keys  are  listed  at  the  bottom  of  the  ANALYSIS  screen: 


• FI  - Help: 

provides  help  on  a number  of  different  ANALYSIS  topics; 

• F2  - Commands: 

lists  all  commands  used  in  ANALYSIS:  you  can  choose  commands 
from  this  screen  to  create  command  statements  for  analysis; 

• F3  - Variables: 

lists  all  of  the  variables  contained  in  your  current  database:  you  can 
choose  variables  from  this  screen  to  create  command  statements  for 
analysis; 

• F4  - Browse: 

allows  you  to  browse  through  the  database; 

• F5  - Printer  on/off:  when  the  printer  is  set  off,  the  results  will  print  on  the  printer:  when 


the  printer  is  set  on,  the  results  will  not  print  to  printer; 

• F9  - DOS: 

allows  you  to  exit  to  DOS  temporarily; 

• FIO  - Quit: 

exits  ANALYSIS  and  returns  to  the  EPI  INFO  Main  Menu. 

Figure  6.  Output  from  TABLES  GENDER  REFERRAL 


GENDER  1 1 2 

M I 4 8 

F 1 4 4 

Total  1 8 12 

REFERRAL 

3 4 5 I Total 

7 3 3 1 25 

7 4 2 1 21 

14  7 5 ! 46 

An  expected  value  is  < 5.  Chi  square  not  valid 
Chi  square  = 1.34 

Degrees  of  freedom  = 4 

p value  = 0.85481294 
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Figure  7.  Output  from  MEANS  AGE  GENDER 


GENDER 


AGE  1 

M 

F ! 

Total 

F 

16  1 

7 

14  1 

21 

17  1 

5 

2 1 

7 

18  1 

2 

3 1 

5 

19  1 

5 

2 1 

7 

20  1 

1 

0 1 

1 

21  1 

4 

0 1 

4 

23  1 

1 

0 1 

1 

F 

Total  1 

25 

21  1 

46 

GENDER 

Obs 

Total 

Mean 

Variance 

Std  Dev 

M 

25 

455 

18.200 

4.250 

2.062 

F 

21 

350 

16.667 

1.133 

1.065 

Difference 

1.533 

GENDER 

Minimum 

25%ile 

Median 

75%ile 

Maximum 

Mode 

M 

16.000 

16.000 

18.000 

19.000 

23.000 

16.000 

F 

16.000 

16.000 

16.000 

17.000 

19.000 

16.000 

ANOVA 

(For  normally  distributed  data  only) 

The 

p value 

is  equivalent 

to  that 

for  the  Student's  T Test, 

since  there 

are  only  2 samples. 

Variation 

SS 

df 

MS  F statistic 

p-value 

Between 

26.833 

1 26. 

833 

9.471  0 

.003856 

Wi  thin 

124.667 

44  2. 

833 

Total 

151.500 

45 

Bartlett's  test  for  homogeneity  of  variance 
Bartlett's  chi  square  = 8.402  deg  freedom  = 1 p-value  = 0.003749 

Bartlett's  Test  shows  the  variances  in  the  samples  to  differ. 

Use  non- parametric  results  below  rather  than  ANOVA. 


Mann-Whitney  or  Wilcoxon  Two-Sample  Test  (Kruskal-Wal I is  test  for  two  groups) 

Kruskal-Wal lis  H (equivalent  to  Chi  square)  = 8.143 

Degrees  of  freedom  = 1 

p value  = 0.004324 

2)  Helpful  Hints  on  Using  ANALYSIS: 

• Instead  of  choosing  commands  and  variables  from  the  F2  and  F3  menus,  the  commands  and 
variables  can  be  typed  in  by  hand.  The  menus  allow  you  to  chose  commands  and  variables 
without  having  to  memorize  or  remember  commands  and  variable  names.  After  you  have 
become  more  experienced  in  using  EPI  INFO  you  may  prefer  to  type  commands  directly 
onto  the  command  line  to  save  time. 
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• Two  tutorials  are  available  in  the  ANALYSIS  program.  They  can  be  accessed  by  typing 
RUN  TUTORl  or  RUN  TUTOR2  in  response  to  the  EPI>  prompt.  For  example: 

EPI>  RUN  TUTORl 

• The  LIST  command  can  be  executed  in  several  ways: 

LIST  prints  out  as  many  variables  as  will  fit  across  the  screen; 

LIST*  prints  out  all  variables  for  all  records; 

LIST  A B will  print  out  variables  A and  B for  all  records;  and 

LIST  NOT  A B will  print  out  all  variables  except  A and  B for  all  records. 

• The  MEANS  command: 

MEANS  AGE  calculates  the  average  AGE; 

MEANS  AGE  GENDER  calculates  the  average  AGE  for  each  category  of  GENDER; 
and 

MEANS  AGE  REFERRAL  GENDER  calculates  the  average  AGE  for  each  category 
of  REFERRAL  within  each  category  of  GENDER. 

When  more  than  one  variable  is  named  in  the  MEANS  command,  the  first  variable  is 
a continuous  variable  and  subsequent  variables  should  be  categorical  variables.  In 
addition,  the  categorical  variables  that  are  selected  should  be  ordered  from  the  variable 
with  the  most  number  of  categories  to  the  variable  with  the  least  number  of  categories. 

Using  the  /N  parameter  following  a MEANS  command  will  withhold  the  display  of  the 
frequencies  table,  but  show  all  other  statistics  associated  with  the  MEANS  analyses  (see 
p.152). 

• The  following  describes  some  data  manipulation  commands  that  can  be  used  to  select 
specific  cases  for  analyses,  organize  the  data,  or  compute  new  variables.  In  using  any  of 
these  commands,  the  statements  will  remain  valid  until  a new  file  is  read  or  until  you  exit 
from  the  ANALYSIS  program.  See  the  EPI  INFO  Version  5.0  manual  for  other  more 
advanced  commands  that  can  be  used  to  transform  or  manipulate  data.  As  well,  the  EPI 
INFO  help  menu  (which  is  accessed  by  pressing  FI)  will  provide  more  detailed  descriptions 
of  the  SELECT,  SORT,  RECODE,  DEFINE,  IF... THEN,  IF... THEN... ELSE,  and 
RECODE  commands. 
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1)  SELECT  can  be  used  to  select  specific  records  for  analysis,  for  example, 

SELECT  GENDER  = "MALE"  AND  AGE  < 21 

will  select  only  cases  where  respondents  are  males  under  the  age  of  21.  Typing 
SELECT  without  any  case  specification  cancels  previous  select  commands. 

2)  SORT  can  be  used  to  sort  cases  on  selected  criteria  before  listing  records,  for  example, 

SORT  GENDER  REFERRAL  AGE 

will  sort  the  cases  into  male  and  female  respondents,  and  will  then  sort  the  cases  by 
referral  category,  and  then  by  age. 

3)  DEFINE  can  be  used  to  define  a variable  name  and  field  type  for  a new  variable.  This 
command  is  frequently  used  in  combination  with  IF  and  RECODE  commands  to 
compute  a new  variable  based  on  information  from  other  variables,  or  to  recode 
information,  for  example, 

DEFINE  COMPLETE  <Y> 

defines  a new  variable,  COMPLETE,  as  a Yes/No  variable. 

4)  Values  for  newly  defined  variables  are  conditionally  defined  by  using  IF... THEN  logic 
statements,  for  example, 

DEFINE  COMPLETE  <Y> 

IF  ((ALCOHOL  >=  1)  OR  (DRUGS  > = 1)  OR  (WHYUSE  > = 1)  OR\ 
(HEALTHIER  > = 1)  OR  (VIDEOS  > = 1)  OR  (DISCUSSION  > = 1)  OR 
(HANDOUTS  > = D)  THEN  COMPLETE="Y". 

That  is,  if  a respondent  completed  at  least  one  of  the  ratings  on  the  client  feedback 
questionnaire,  then  the  client  will  be  considered  to  have  completed  the  questionnaire. 

It  is  important  to  cover  all  conditions  when  using  IF  ...  THEN  statements  to  recode  a 
variable. 

The  following  statement  describes  when  a participant  should  be  considered  to  have  not 
completed  the  feedback  questionnaire: 

IF  NOT  ((ALCOHOL  > = 1)  OR  (DRUGS  > = 1)  OR  (WHYUSE  > = 1)  OR\ 
(HEALTHIER  > = 1)  OR  (VIDEOS  > = 1)  OR  (DISCUSSION  > = 1)  OR 
(HANDOUTS  >=  D)  THEN  COMPLETE="N" 
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IF... THEN... ELSE  statements  can  also  be  used  to  cover  all  possible  value  conditions, 
or  example, 

DEFINE  COMPLETE  <Y> 

IF  ((ALCOHOL  > = 1)  AND  (DRUGS  > = 1)  AND  (WHYUSE  > = 1)  AND\ 
(HEALTHIER  > = 1)  AND  (VIDEOS  > = 1)  AND  (DISCUSSION  > = 1)  AND 
(HANDOUTS  >=  D)  THEN  COMPLETE="Y"  ELSE  COMPLETE  = ' N", 

defines  a respondent’s  status  on  the  feedback  questionnaire  as  complete  only  if  all 
program  component  ratings  were  completed,  and  as  incomplete  when  one  or  more 
usefulness  ratings  are  missing. 

5)  RECODE  will  re-code  values  for  an  existing  variable.  Before  recoding  a variable  it 
is  best  to  first  DEFINE  a new  variable  to  store  the  newly  re-coded  data,  for  example, 

DEFINE  NGENDER  # 

RECODE  GENDER  TO  NGENDER  M=1  F=2, 

will  replace  all  "M"  and  "F"  alphanumeric  entries  for  GENDER  to  I’s  and  2’s  which 
are  stored  in  the  new  variable  NGENDER. 

The  next  example  shows  how  age  group  categories  can  be  computed  using  RECODE: 

DEFINE  AGEGRP  # 

RECODE  AGE  TO  AGEGRP  16-19=YOUNGER  20-23  = OLDER. 

6)  Arithmetic  expressions  can  also  be  used  to  create  values  for  newly  defined  variables. 
For  example, 

DEFINE  OVERALL 

OVERALL=  (ALCOHOL  -f  DRUGS  -f  WHYUSE  -1-  HEALTHIER  + \ 
VIDEOS  + DISCUSSION-f  HANDOUTS)/? 

computes  values  for  the  newly  defined  overall  rating  variable,  with  scores  that  range 
from  1 to  5. 
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Appendix  B.  Description  of  Lotus  1-2-3 


Lotus  1-2-3  is  a spreadsheet  software  that  includes  the  capacity  to  generate  basic  statistics  as  well 
as  graphical  display  of  information.  More  information  on  Lotus  can  be  obtained  from  Lotus 
Development  Canada,  Ltd.,  10  Bay  Street,  Toronto,  Ontario,  M5J  2R8,  or  by  calling  1-800-GO- 
LOTUS. 
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Appendix  C.  Description  of  Harvard  Graphics 


Harvard  Graphics  is  a graphics  software  package  that  includes  the  capacity  for  making  a broad 
range  of  charts,  graphs  and  other  display  formats.  More  information  on  Harvard  Graphics  can 
be  obtained  from  SPG  Software  Publishing,  1595  Sixteenth  Avenue,  Suite  303,  Richmond  Hill, 
Ontario,  L4B  3N9. 


The  Addiction  Research  Foundation  offers  a wide  variety  of  materials  that  can  help  you  assess 
and  treat  addictions,  including: 


The  Directory  of  Outcome  Measures  for  Addictions  Treatment  Programs. 

$39 

This  directory  can  help  you  to  identify  the  most  useful  and  effective  ways  to 
monitor  addictions  treatment.  It  includes  information  on  more  than  25  measures 
designed  to  assess  client  outcomes  regarding  substance  abuse,  other  life  areas 
such  as  depression  and  self-esteem,  and  client  satisfaction.  Each  entry  includes 
information  on  target  population,  administrative  reliability,  scientific  validity  and 
ability  to  measure  change. 

Alcohol  & Drug  Problems:  A Practical  Guide  for  Counsellors 
$34.95 

Written  by  experts  in  the  fields  of  social  work,  psychology,  law  and  medicine, 
this  book  offers  “hands-on”  information  to  anyone  whose  clients  may  have 
alcohol  and  other  drug  problems.  A wide  range  of  topics  is  explored,  including: 
ethics  and  professional  issues;  relapse  prevention;  dual  disorders;  dealing  with 
special  populations  such  as  physically  disabled  clients,  lesbians  and  gay  men;  and 
related  issues,  such  as  working  with  probation  and  parole  clients,  or  people  with 
drug  and  alcohol  problems  who  are  HIV-positive. 


To  order,  or  for  more  information  on  these  and  other  products  available  through  the  Addiction 
Research  Foundation: 

Please  call  the  ARF  Marketing  Department  at  1-800-661-1111 
or 

Write  to: 

Addiction  Research  Foundation 
Marketing  Department 
33  Russell  Street 
Toronto,  Ontario 
M5S  2S1 
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The  Evaluation  Casebook 


Using  Evaluation  Techniques 
to  Enhance  Program  Quality 
in  Addictions 
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